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ABSTRACT 


This  thesis  research  combines  several  proven  methods  by  which  public 
participation  can  be  used  more  effectively  in  a  government  decision-making  process. 
The  research  involved  fulfills  three  primary  purposes.  First,  the  research  provides  a 
flexible  user-friendly  internet-based  platform,  whereby  the  knowledge  level  of  a  disparate 
group  of  stakeholders  can  be  improved  with  respect  to  a  complex  technical  subject. 
Second,  the  research  demonstrates  a  method  by  which  stakeholder  consensus  is  derived. 
Third,  the  research  exhibits  a  method  by  which  public  values  are  aggregated,  whatever 
the  level  of  consensus;  the  data  is  then  provided  to  the  government  for  use  in  a  decision¬ 
making  model. 
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I.  INTRODUCTION 


A.  BACKGROUND 

Since  the  end  of  the  Cold  War,  the  United  States  has  experienced  a  significant 
reduction  of  military  forces  and  bases.  The  Base  Realignment  and  Closure  (BRAC) 
commission  was  established  in  1993  for  a  single  purpose:  “to  quickly  dispose  of  excess 
activities,  as  designated  for  closure  or  realignment  by  the  U.S.  Congress,  to  local 
communities  for  reuse  and  economic  revitalization.”  (NAVFAC,  1999)  The  base 
closures  not  only  burden  the  local  government  and  its  people  with  decreased  revenue  and 
increased  unemployment;  but,  the  closures  leave  contaminants  behind,  where  the  polluted 
lands  and  waters  increase  the  risk  of  public  health  and  safety.  In  1984,  Congress 
established  the  Defense  Environmental  Restoration  Program  (DERP)  to  evaluate  and 
remedy  contamination  from  past  Department  of  Defense  (DOD)  activities,  and  to  protect 
human  health  and  environment  from  risks  associated  with  contamination  of  DOD  lands. 
DERP  was  originally  funded  by  DOD  Environmental  Accoimt  (DERA),  whose  purpose 
is  to  provide  funding  for  cleaning  at  operational  installations  and  formerly  used  defense 
sites.  As  a  result  of  BRAC  decisions,  the  number  of  restoration  sites  has  grown 
dramatically,  having  doubled  since  1989  to  approximately  4500  sites  today.  More 
specifically,  the  current  number  of  U.S.  Navy  activities  slated  for  closure  is  178  and  the 
property  at  91  of  these  activities  has  been  or  is  slated  for  disposal.  (NAVFAC,  1999)  The 
administrative  burden  for  the  overwhelming  number  of  closure  and  cleanup  projects  has 
grown  correspondingly.  DOD  responded  to  this  gro-wth  by  devolving  DERA  into 
separate  Army,  Navy,  and  Air  Force  restoration  programs.  The  DOD  now  serves  as  a 
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coordinator  for  the  services,  and  each  service  is  given  the  responsibility  of  managing  its 
program  more  effectively.  Breaking  the  organization  into  components  also  allows  each 
office  to  be  more  responsive  to  stakeholders  of  any  given  project. 

The  naval  component,  known  as  the  Department  of  the  Navy’s  Environmental 
Restoration  Program  (ERP,  N)  is  the  responsibility  of  the  Office  of  the  Assistant 
Secretary  of  the  Navy  for  Installations  and  Environment  (ASN  (I&E)).  Through  the 
office  of  the  Chief  of  Naval  Operations  (CNO),  the  Naval  Facilities  Engineering 
Command  (NAVFAC)  manages  all  shoreside  environmental  projects  and  programs, 
providing  technical  expertise  to  support  the  Navy’s  environmental  initiatives  and 
coordinates  with  legislative  and  regulatory  agencies.  NAVFAC  also  manages  a  natural 
resource  program  to  enhance  the  environmental  attributes  of  its  land,  forests  and  wildlife. 
(NAVFAC,  1999)  Ten  engineering  field  divisions  and  engineering  field  activities, 
located  across  the  United  States  and  Europe,  provide  engineering  support  and  services  to 
the  naval  shore  establishment.  The  Engineering  Field  Activity,  West  (EFA,  West)  serves 
as  the  NAVFAC  regional  headquarters  for  Central  and  Northern  California,  as  well  as 
Nevada. 

EFA,  West  officials  face  the  challenge  of  choosing  remediation  technologies  that 
satisfy  several  goals  such  as  reducing  cost  and  decreasing  human  and  ecological  risk, 
while  satisfying  the  public.  The  problem  EFA,  West  faces  is  to  choose  the  best  treatment 
technology  using  public  input.  However,  EFA,  West  is  not  certain  as  to  how  to 
operationalize  public  values.  Including  public  opinion  into  a  decision-making  process 
has  been  a  difficult  task  for  government  officials  because  of  several  barriers  that  exist  for 
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utilizing  public  input  such  as:  geographic  distance,  logistics  cost,  available  time,  and  a 
structured  model  for  aggregating  input. 

B.  PURPOSE 

This  thesis  research  is  one  of  a  few  which  combines  several  proven  methods  by 
which  public  participation  can  be  used  more  effectively  in  a  government  decision-making 
process.  The  research  involved  fulfills  three  primary  purposes.  First,  the  research 
provides  a  flexible  user-fiiendly  internet-based  platform,  whereby  the  knowledge  level  of 
a  disparate  group  of  stakeholders  can  be  improved  with  respect  to  a  complex  technical 
subject.  Second,  the  research  demonstrates  a  method  by  which  stakeholder  consensus  is 
derived.  Third,  the  research  exhibits  a  method  by  which  public  values  are  aggregated, 
whatever  the  level  of  consensus;  the  data  is  then  provided  to  the  government  for  use  in  a 
decision-making  model. 

The  method  used  in  this  thesis  research  may  be  useful  to  any  organization  which 
must  satisfy  a  heterogeneous  collection  of  stakeholders  with  different  values  and 
interests.  The  methodology  addresses  the  commonplace  problem  of  having  to  aggregate 
the  public’s  values  after  deriving  as  much  consensus  as  possible  among  the  participating 
members.  The  method  may  help  any  organization  interested  in  utilizing  the  concerns  of 
stakeholders  as  an  integral  part  of  the  decision-making  process. 

C.  METHODOLOGY 

The  methodology  used  in  this  thesis  research  consists  of  the  following: 

•  Conduct  a  literature  review  of  research  on  various  decision-making  and 
statistical  methods  including  Multiattribute  Utility  Theory  (MAUT),  Value 
Tree  Analysis  (VTA),  Analytical  Hierarchy  Process  (AHP),  Delphi 
method,  and  various  aggregation  methods. 
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•  Research  characteristics  of  Restoration  Advisory  Boards  (RABs)  from  the 
following  sites:  Naval  Air  Station  (NAS)  Alameda,  Concord  Naval 
Weapons  Station,  Mare  Island  Naval  Shipyard,  NAS  Moffet  Field, 
Treasure  Island  Naval  Station,  Treasure  Island  Naval  Station-Himters 
Point  Annex,  Oakland  Fleet  and  Industrial  Supply  Center  (FISC).  Utilize 
the  characteristics  to  assign  the  roles  to  proxy-RAB  members. ' 

•  Develop  a  discussion  forum  for  the  discussion  phase  of  the  Delphi 
method,  where  stakeholders  can  voice  their  opinions  and  educate  others  on 
the  consequences  of  various  remedy-technology  alternatives.  Readily 
available  and  free  commercial  software  is  used  to  develop  the  forum. 

•  Ask  the  proxy-RAB  panel  to  use  the  forum  to  express  their  site-specific 
concerns  (i.  e.  values)  regarding  remedy  technologies.  In  each  of  two 
Delphi  rounds,  the  panel  is  asked  to  rank  the  set  of  values  in  order  of 
importance.  The  level  of  association  between  the  two  rounds  is 
determined  through  the  used  of  simple  aggregation  methods  and 
correlation  determination.^. 

•  Publish  the  results  of  the  rankings  on  the  web  after  each  round  of  Delphi 
for  further  discussion  and  ranking. 

•  Determine  the  proxy-RAB  members’  representative  final  ordinal  ranking 
of  the  set  of  alternatives  using  an  aggregation  technique. 

•  Convert  final  ordinal  ranking  into  attribute  weights. 

•  Construct  the  completed  value  tree. 

D.  JUSTIFICATION  FOR  THE  STUDY 

This  thesis  research  involves  the  integration  of  proven  decision-making  methods, 
namely  the  Delphi  method  and  VTA  using  MAUT,  into  an  internet-based  platform. 
Traditionally,  the  Delphi  method  is  performed  by  sending  questionnaires  to  stakeholders 
(RABs),  or  organizing  a  conference  at  a  specific  location.  This  thesis  research  uses  an 


Due  to  time  constraints  of  research,  volunteers,  none  of  them  belong  to  RABs,  study  the  characteristics 
of  RABs  and  subsequently  role  play  for  genuine  RAB  members. 

2 

Originally,  the  Median  Ranking  method  and  the  Hungarian  method  were  used  measure  the  degree  of 
divergence  (or  convergence)  of  the  proxy-RAB  members’  concerns  (i.e.  values)  at  the  end  of  each  round. 
Efforts  to  derive  an  assignment  matrix  failed  due  to  the  data  size;  and  as  a  result,  simpler  methods  of 
aggregation  and  correlation  were  used  to  legitimate  the  consensus  derived  in  developing  the  value  tree. 
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internet-based  platform  to  eliminate  the  logistic  barriers  such  as  costs  involved  with 
administration  and  logistics  and  time  for  all  stakeholders  to  gather.  The  mean  value, 
standard  deviation  and  correlation  coefficients  are  calculated  to  determine  the  level  of 
association  between  stakeholders’  opinions,  and  final  consensus. 

E.  RESEARCH  QUESTIONS 

The  primary  research  question  in  this  thesis  research  is: 

What  is  a  pragmatic  and  theoretically  sound  method  to  gather  the  public’s  opinion 
into  a  rigorous  decision-making  process  by  using  information  technology? 

The  secondary  research  questions  are: 

•  Could  this  methodology  be  useful  in  explaining  the  many  considerations 
in  government  decisions? 

•  Can  the  Delphi  be  implemented  on  the  Internet? 

•  How  can  a  technology  choice  problem  be  stmctured  by  using  decision¬ 
making  methods  such  as  Delphi,  MAUT,  VTA  or  AHP? 

F.  RESEARCH  SCOPE  AND  LIMITATIONS 

The  scope  of  this  thesis  is  to  develop  a  web-based  communications  structure  for 
stakeholders  to  voice  their  opinions  anonymously,  and  rank  their  preferences  according  to 
their  specific  values  after  using  the  commimication  structure  to  learn  more  about  a 
complex  and  technical  subject.  Moreover,  this  research  combines  the  various  decision¬ 
making  methods  to  reach  a  group’s  decision  output,  which  will  be  given  to  EFA,  West 
officials  to  decide  to  adopt  the  methodology  used  in  this  research. 

Although  the  research  provides  a  valuable  demonstration  of  combining  decision¬ 
making  techniques  with  the  Internet,  the  research  has  some  limitations.  First,  proxy-RAB 
members  substituted  for  real  RAB  members  due  to  time  constraints.  It  is  impossible  to 
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know  for  certain  whether  the  research  results  would  be  different  using  real  RAB 
members.  An  important  factor  to  consider  is  that  proxy-RAB  members  do  not  have  a 
genuine  and  long-held  understanding  of  the  concerns  that  are  particular  to  the  actual  sites 
they  are  representing.  Second,  time  constraint  also  limited  the  amount  of  time  for 
discussion  during  Delphi  rounds.  Furthermore  the  number  of  Delphi  roimds  is  limited  to 
two,  whereas  normally  three  or  more  are  recommended. 

G.  OVERVIEW  AND  ORGANIZATION 

This  thesis  is  comprised  of  five  chapters.  This  chapter  provides  background 
concerning  EFA,  West’s  involvement  in  the  San  Francisco  Bay  Sediment  Project.  The 
research  describes  the  use  of  an  electronic  internet-based  forum,  where  stakeholders  can 
voice  their  opinions  and  prioritize  their  concerns  vis-a-vis  remedy  technologies. 

Chapter  II  is  the  review  of  the  literature,  which  explains  various  decision-making 
and  statistical  techniques  including  MAUT,  VTA,  AHP,  the  Delphi  Technique,  the 
Median  Ranking  Method,  the  Hungarian  Method,  and  various  aggregation  methods. 
Chapter  III  discusses  the  formulation  and  implementation  of  an  internet-based  discussion 
forum  that  allows  geographically  distant  proxy-RABs  to  list  their  important  concerns,  and 
rank  the  set  of  attributes  anonymously.  Chapter  IV  provides  the  research  results,  and 
evaluates  the  efficiency  of  the  internet,  and  provides  recommendations  for  developing 
new  methodologies  to  accomplish  the  purpose  of  this  thesis. 

H.  SUMMARY 

For  decades,  the  government  has  encouraged  public  participation;  however, 
organizations  do  not  have  effective  models  to  incorporate  public  opinion.  The  Delphi 
method  allows  the  public  to  participate  in  a  decision-making  process,  where  the  public 
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voices  its  concerns  and  ranks  its  attributes  anonymously.  Delphi  has  proven  itself  as  a 
systematic  and  effective  way  of  involving  public  values.  This  research  investigates 
whether  Delphi  can  be  combined  with  VTA  and  MAUT  using  Internet-based  elicitation 
techniques.  This  research  creates  a  bridge  that  allows  government  agencies  to  use  such 
methodology  in  future  decision-making  practices. 
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II.  LITERATURE  REVIEW 


A.  INTRODUCTION 

This  chapter  reviews  various  types  of  evaluation  and  decision-making  methods. 
The  purpose  is  to  furnish  readers  with  a  backgrotmd  of  selected  decision-making 
methods,  which  will  provide  the  necessary  tools  to  utilize  stakeholder’s  values  in  the 
decision-making  process.  This  chapter  is  divided  into  three  sections:  1)  Multi-criteria 
decision  making  techniques,  2)  Value  solicitation  and  consensus  building  methods,  and 
3)  Aggregation  methods.  Multi-criteria  decision-making  techniques,  specifically  Value 
Tree  Analysis  (VTA)  utilizing  Multi-attribute  Utility  Theory  (MAUT),  and  a  competing 
method,  Analytic  Hierarchy  Process  (AHP)  are  investigated,  where  simply  stated 
procedures  and  applications  are  provided.  Value  solicitation  and  consensus  building 
focus  on  the  basics  of  the  Delphi  method,  the  consensus  building  technique  used  in  this 
thesis.  An  application  of  the  Delphi  using  information  technology  is  demonstrated  in 
Chapter  III.  The  third  section  lists  various  types  of  aggregation  techniques,  and  explains 
the  importance  of  aggregation  when  there  are  various  sizes  of  groups  of  experts  that  need 
to  make  a  group  decision. 

B.  BACKGROUND  -  ASPECTS  OF  DECISION-MAKING 

Figure  1  illustrates  two  major  aspects  involved  in  the  decision-making  process. 
The  first  is  Procedural  and  Technological  Interventions,  where  well-developed  models 
and  guidelines  are  introduced  to  help  facilitate  the  decision-making  process.  The  second 
aspect  encompasses  the  major  features  of  decision-making,  which  explains  how  a 
problem  evolves  and  the  need  for  understanding  the  problem  before  making  a  decision. 
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While  the  authors  give  no  formal  title,  this  thesis  will  reference  this  aspect  as  Problem 


Structuring  and  Legitimation. 


Figure  2.1.  Aspects  of  decision-making  (Kleindorfer,  Kunreuther  and 
Schoemaker,  1993).  Note:  Changes  have  been  made  to  original  to  reflect  the  relation 
between  the  two  aspects  (Procedural  and  Technological  Interventions  and  Problem 

Structuring  and  Legitimation). 
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The  two  aspects  explained  above  are  a  product  of  prescriptive  and  descriptive 
theories  based  on  research  by  Kleindorfer,  Kunreuther  and  Schoemaker  (1993).  While 
both  theories  address  the  “how”  of  a  decision,  descriptive  analysis  examines  the  human 
quest  for  making  decisions,  while  the  prescriptive  aspect  defines  measures  upon  which  a 
decision  should  be  based.  Consider  a  group  decision  situation.  An  illustration  of 
descriptive  theory  in  group  decisions  would  be  sociology,  which  “provides  a  perspective 
on  the  nature  of  information  diffusion  processes  and  the  role  of  social  norms  in  the 
decision-making  process.”  (Kleindorfer,  Kunreuther  and  Schoemaker,  1993,  p.  5)  Game 
Theory,  a  prescriptive  example,  focuses  on  competition,  and  is  defined  as  “the  study  of 
how  optimal  strategies  are  formulated  in  conflict.”  (Brown,  1998,  p.  31)  The  following 
paragraphs  explain  the  need  for  both  aspects,  and  the  interdependence  that  must  exist  for 
a  sound  decision-making  process. 

1.  Procedural  and  Technological  Interventions 

Procedural  and  Technological  Interventions  provide  structured  guidelines  for 
individuals  or  groups  who  must  make  a  decision.  In  operations  analysis,  models  are 
developed  to  answer  the  well-known  economic  tradeoff  problem  of  maximizing  profit 
while  reducing  risk.  Technology  plays  a  significant  role  by  introducing  systems  that  help 
decision-makers  evaluate  complex  problems. 

Guidelines  and  models  offer  prescriptive  solutions  to  a  decision-maker;  but  at  the 
same  time,  lack  the  human  intervention  that  must  be  considered.  The  need  for  conflict 
resolution  among  decision-makers  with  different  goals  and  objectives  helps  define  the 
organization’s  structure  and  decision-making  process.  This  void  of  the  human  psyche 
warrants  the  need  for  the  descriptive  aspect  of  evaluating  complex  problems. 
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2.  Problem  Structuring  and  Legitimation 

In  Figure  1,  Problem  Structuring  and  Legitimation  describe  some  features 
involved  in  descriptive  theory  and  is  broken  into  four  major  categories:  Problem  context, 
problem  finding,  problem  solving  and  legitimation  process.  (Kleindorfer,  Kunreuther  and 
Schoemaker,  1993)  Problem  context  defines  the  social,  institutional  and  informational 
environment  of  the  decision  process.  Understanding  problem  context  means  recognizing 
who  might  be  affected,  who  has  power  (or  final  authority),  what  information  is  available 
to  the  decision-makers,  and  if  possible,  what  values  or  beliefs  they  hold  regarding  the 
problem  in  question. 

Problem  finding  addresses  the  fact  that  a  problem  has  been  identified  by  the 
activity,  accepted  by  the  decision-maker(s),  and  represented  to  give  them  a  decision 
opportunity.  Problem  solving  involves  making  a  choice  that  resolves  the  presented 
problem.  This  choice  is  based  on  understanding  the  relevant  values  and  identifying 
solution  alternatives.  The  Legitimation  process  consists  of  determining  how  the 
stakeholders  will  be  affected  by  the  chosen  solution  alternative,  and  rationalizing  the 
choice  of  the  decision-maker  in  terms  of  stakeholder  values  and  interests.  (Kleindorfer, 
Kunreuther  and  Schoemaker,  1993) 

The  upward  arrow  between  Procedural  and  Technological  Interventions  and 
Problem  Structuring  and  Legitimation  (see  Figure  1)  represents  the  need  for  a  problem  to 
be  well  defined  and  imderstood  by  the  decision-makers  before  utilizing  a  decision¬ 
making  process.  Kleindorfer,  Kunreuther  and  Schoemaker  state  that  “soxind  prescription 
must  begin  with  good  description.”  (1993,  p.  10)  The  arrow’s  downward  direction 
indicates  that  the  use  of  specific  types  of  Procedural  and  Technological  Interventions  may 
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help  the  decision-maker  better  understand  and  define  the  problem.  If  the  desired  outcome 
is  not  obtained,  it  is  because  of  ill-defined  factors  that  affect  the  overall  problem.  The 
arrows  connecting  the  categories  in  the  second  aspect  indicate  there  are  no  set  procedures 
of  which  category  should  be  resolved  first.  Rather,  it  is  a  redundant  or  cyclical  process, 
where  one  change  in  one  category  affects  the  other  three  categories.  Only  when  the 
decision-maker  imderstands  all  four  categories  in  the  aggregate  can  the  problem  be 
considered  well  defined,  and  only  then  can  the  proper  tools  be  selected  or  designed  to 
help  the  decision-maker  make  decisions. 

C.  MULTI-CRITERIA  DECISION  MAKING  TECHNIQUES 

Questions  like  “Is  this  plan  wise?”  “Should  I  choose  option  A  or  option  B?” 
“How  well  is  this  program  doing?”  have  been  asked  about  social  programs  for  decades 
(Edwards  and  Newman,  1982).  But  the  idea  that  one  could  answer  such  questions 
systematically  and  in  a  manner  other  than  by  simply  making  an  intuitive  judgment,  is  a 
product  of  decision  science  research  since  the  1950s  and  1960s.  Organizations  strive  to 
work  more  efficiently  by  choosing  programs  worthy  of  support  over  those  that  are  not, 
and  providing  guidance  for  programs  in  existence.  At  the  same  time,  organizations 
devote  resources  and  attention  to  develop  satisfying  methods  to  improve  efficiency  in 
programs. 

1.  Multi- Attribute  Utility  Theory  (MAUT) 

a.  Definition  of  MAUT 

Multi-attribute  Utility  Theory  (MAUT)  is  a  tool  used  to  structure  a 
decision-maker’s  preferences.  MAUT  presents  the  approach  to  determine  the  priorities 
of  the  decision-maker,  and  is  a  tool  in  which  organizations  can  organize  and  aggregate  its 
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evaluative  efforts.  MAUT  applies  to  situations  with  uncertainty  and  multiple,  often 
conflicting  objectives  (Murphy,  1997).  According  to  Edwards,  Guttentag,  and  Snapper 
(1975),  MAUT  depends  on  a  few  key  ideas: 

•  When  possible,  evaluations  should  be  comparative. 

•  Programs  normally  serve  multiple  constituencies  (those  that  have  the 
power  to  vote/elect). 

•  Programs  normally  have  multiple  goals,  not  all  equally  important. 

•  Judgments  are  inevitably  a  part  of  any  evaluation. 

•  Judgements  of  magnitude  are  best  when  made  numerically. 

•  Evaluations  typically  are,  or  at  least  should  be,  relevant  to  decisions. 
Evaluations  of  programs  that  serve  multiple  constituencies  should 

normally  be  addressed  to  the  interests  of  those  constituencies.  Different  constituencies 
can  be  expected  to  have  different  interests.  If  programs  have  multiple  goals,  evaluations 
should  assess  how  well  the  programs  serve  the  constituencies  and  the  programmatic 
goals;  this  implies  multiple  measures  and  comparisons.  Edwards  and  Newman  state  that 
using  “multiple  measures  of  effectiveness  (which  may  well  be  simple  subjective 
judgments  in  munerical  form)  makes  less  appealing  the  notion  of  social  programs  as 
experiments.”  (1982,  p.  8)  However,  implementation  of  such  experiments  become  more 
difficult  as  the  number  of  measures  needed  for  a  satisfactory  evaluation  increases. 
Experimental  or  hard  data  can  be  easily  incorporated  in  a  MAUT  evaluation,  provided 
such  data  are  available. 

Edwards  and  Newman  (1982)  suggest  that  combining  subjectivity  with 
evaluation  with  numerical  judgements  serve  several  useful  purposes: 
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•  First,  the  combination  partly  closes  the  gap  between  intuitive  and 
judgmental  evaluations,  and  makes  coexistence  of  judgement  and 
objective  measurement  within  the  same  evaluation  easy  and  natural. 

•  Second,  accepting  subjectivity  into  evaluation  provides  an  opportunity  to 
combine  links  of  values.  For  instance,  “evaluation  researchers  often 
distinguish  between  process  evaluations  and  outcome  evaluations.” 
(Edwards  and  Newman,  1982,  p.  9)  Process  and  outcome  are  different, 
but  if  a  program  has  goals  of  both  kinds,  its  evaluation  can  and  should 
assess  its  performance  on  both. 

•  Third,  the  use  of  subjective  inputs  can  reduce  the  time  required  for  an 
evaluation  to  be  executed.  In  as  little  as  a  week  of  concentrated  effort,  a 
MAUT  evaluation  can  be  used  from  its  original  definition  of  the 
evaluation  problem  to  the  preparation  of  the  evaluation  report. 

b.  Purposes  for  Evaluations 

According  to  Edwards  and  Newman,  MAUT  is  a  “widely  applicable 
method  of  organizing  and  presenting  evaluative  information.  As  such,  it  is  compatible 
with  any  other  evaluative  activity  designed  to  yield  numbers  as  outputs.”  (1982,  p.  11) 
The  organizational  requirement  for  evaluation  is  normally  based  on  the  supposition  that 
decisions  need  to  be  made.  Sometimes,  the  decisions  involve  programmatic  continuation, 
modification  or  elimination.  Other  decisions  might  revolve  around  program  design, 
management,  or  process  improvement. 

Major  evaluations  are  often  required  as  a  basis  for  potential  major 
programmatic  changes  -  up  to  and  including  the  most  major  of  all  changes:  the  birth  or 
death  of  a  program.  Funding-level  decisions  are  also  programmatic  choices;  the  same 
program  at  two  substantially  different  funding  levels  results  in  qualitatively  different 
programs. 


The  aforementioned  decisions  share  two  characteristics.  The  first  is  that 
all  evaluations  require  comparison  of  something  with  something  else.  An  important 


15 


implication  of  the  comparative  nature  of  virtually  every  evaluation  is  that  some  of  the 
comparisons  are  inevitably  between  the  program  as  it  is  and  the  program  as  it  might  be  — 
that  is,  between  the  existing  and  the  conceptual  programs  or  programmatic  methods.  The 
second  is  that  programs  virtually  always  have  multiple  objectives;  consequently, 
evaluations  should  assess  as  many  objectives  as  possible.  MAUT  serves  as  a  useful  tool 
in  these  cases  because,  as  mentioned  before,  MAUT  is  directed  at  analyzing  problems 
with  multiple  goals  serving  multiple  constituencies. 

c.  Steps  in  a  MA  UT  Evaluation 

In  many  decision  theory  books,  authors  attempt  to  define  the  pertinent 
steps  that  are  used  to  execute  the  MAUT  evaluation.  Some  have  stated  five  steps,  others 
more.  To  define  each  step  clearly,  the  process  is  broken  into  seven  steps,  as  used  by 
Edwards  and  Newman  (1982): 

•  Step  one:  Identify  the  objects  of  evaluation  and  the  function  or  functions 
that  the  evaluation  is  intended  to  perform.  The  functions  of  the  evaluation 
will  often  control  the  choice  of  objects  of  evaluation. 

•  Step  two:  Identify  the  stakeholders.  Stakeholders  are  defined  as  “people 
who  have  an  interest  or  stake  in  the  program  or  entity  being  evaluated,  and 
who  are  important  enough  so  that  their  interests  should  be  considered.” 
(Edwards  and  Newman,  1982,  p.  17) 

•  Step  three:  Elicit  the  relevant  value  dimensions  or  attributes  from 
stakeholder  representatives,  and  organize  the  value  dimensions  or 
attributes  into  a  hierarchical  structure. 

•  Step  four:  Assess  the  relative  importance  of  each  of  the  values  identified 
in  step  three  for  each  stakeholder  group.  Such  judgements  can  be 
expected  to  vary  firom  one  stakeholder  group  to  another;  methods  of 
addressing  such  value  conflicts  are  important. 

•  Step  five:  Ascertain  how  well  each  object  of  evaluation  (i.e.  alternative) 
meets  the  values  implicit  in  each  attribute.  To  understand  the  level  of  any 
alternative’s  success  in  meeting  an  attribute  requires  measurements  or 
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subjective  expert  judgements,  or  both.  These  measures  and  judgements 
must  be  converted  into  a  common,  comparable  scale,  called  utility. 

Step  six:  Aggregate  utilities  with  measures  of  importance.  This  is  done 
with  the  general  equation: 


n 

Uj  =  SwjUjj 

i  =  l 


Where  Uj  is  the  overall  or  composite  utility  for  the  j*^  decision  alternative; 
Wi  is  the  normalized  weight  assigned  to  the  i*  attribute;  and  uy  is  the  utility 
of  the  j*  option  on  the  i*  attribute.  The  symbol  E  means  to  sum  the 
weighted  utilities  over  all  the  attributes  from  the  first  (1)  to  the  last  (n). 
The  larger  the  numerical  value  of  U,  the  better.  Therefore,  whatever 
decision  alternative  received  the  largest  U  should  be  considered  best 
among  the  alternatives. 

•  Step  seven:  Perform  sensitivity  analyses.  The  question  underlying  any 
sensitivity  analysis  is  whether  a  change  in  the  analysis  will  lead  to 
different  conclusions,  for  example,  using  different  weights  in  the  utility 
measurement.  While  conclusions  may  have  emerged  from  step  six,  the 
conclusions  deserve  credence  as  a  basis  for  action  only  after  their 
sensitivity  is  explored  in  this  step. 

d.  The  Relation  Between  Evaluation  and  Decision 

Researchers  have  been  concerned  with  the  relationship  between  evaluation 

and  decision,  especially  the  academic  evaluation  researchers,  who  wonder  whether  or  not 

their  evaluations  are  used  appropriately.  Some  evaluators  believe  their  responsibility  is 

to  provide  the  relevant  facts;  and  someone  else  should  make  the  decisions.  As  a  rule,  the 

evaluator  is  not  the  decision-maker,  and  cannot  compel  the  decision-maker  to  agree  with 

the  result  of  the  evaluation,  or  to  base  decisions  on  it. 

Edwards  and  Newman  (1982)  explain  three  devices  that  make  evaluations 

more  likely  to  be  used  in  decision-making.  The  first  and  most  important,  is  to  involve  the 

decision-makers  heavily  in  the  evaluative  process;  especially  when  the  decision-makers 
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are  among  the  most  important  stakeholders.  The  second  is  to  make  the  evaluation  as 
directly  relevant  to  the  decision  as  possible,  preferably  by  making  sure  that  the  options 
available  to  the  decision-maker  are  the  objects  of  evaluation.  The  third  is  to  make  the 
product  of  the  evaluation  useful  -  which  primarily  means  making  it  imderstandable.  In 
this  research,  the  proxy-RAB  members  (representing  stakeholders)  identify  the 
parameters  on  which  evaluators  should  focus.  Additionally,  the  decision-makers  use  the 
stakeholder-defined  value  stracture  to  make  decisions. 
e.  Summary 

This  research  conforms  to  the  recommendations  of  Edwards  and  Newman 
by  ensuring  that  decision-makers  are  involved  in  the  evaluation  process;  by  making 
evaluation  as  directly  relevant  to  the  decision  as  possible;  and  by  making  the  product  of 
the  evaluation  understandable.  The  latter  point  is  particularly  important  in  a  publicly 
funded  program  that  depends  upon  constituent  satisfaction.  Satisfaction,  in  turn,  can  only 
be  produced  by  legitimating  the  decisions.  In  this  spirit,  the  method  adheres  to  the 
requirements  needed  to  use  the  simple  additive  utility  as  depicted  in  Section  C.l(c).  The 
requirements  for  using  the  Additive  Utility  Function,  and  a  further  description  of  this 
method,  are  included  in  Appendix  A. 

2.  Value  Tree  Analysis  (VTA) 

Complex  evaluation  problems  can  be  structured  by  building  a  value  tree 
(Edwards,  1980).  Value  trees  are  pyramid-like  structures.  At  the  top  of  the  tree  is 
generally  one  overall  value.  “Values  are  abstractions  that  help  organize  and  guide 
preferences.  They  are  most  often  expressed  as  statements  of  desired  states,  positive 
intentions,  or  preferred  directions.  The  actions  or  objects  of  value  may  be  such  diverse 
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choices  as  social  policies,  marketing  strategies,  or  individual  consumer  choices.” 
(Winterfeldt  and  Edwards,  1986,  p.  38)  Values  are  also  defined  as  a  collection  of 
objectives.  Clemen  states,  “An  individual’s  objectives  taken  together  make  up  his  or  her 
values.  They  define  what  is  important  to  that  person  in  making  a  decision.  A  person’s 
values  are  the  reason  for  making  decisions  in  the  first  place.  Without  objectives,  it  would 
not  be  possible  to  tell  which  alternative  would  be  the  best  choice.”  (1996,  p.  20) 

Below  the  overall  goal  are  several  intermediate  level  goals,  or  objectives,  needed 
to  reach  the  overall  goal.  Intermediate  goals  can  include  fimdamental  and  means 
objectives.  Distinguishing  between  fimdamental  and  the  means  objectives  allows  the 
decision-maker  to  structure  the  value  tree.  “Fundamental  objectives  reflect  what  we 
really  want  to  accomplish  and  means  objectives  help  achieve  other  objectives. 
Stracturing  the  fimdamental  objective  hierarchy  is  crucial  for  developing  a  multiple 
objective  decision  model.”  (Clemen,  1995  p.  44) 

At  the  lowest  level  are  the  attributes,  “which  serve  to  explicate  and  operationalize 
higher  level  values  and  concerns.”  (Stillwell,  Winterfeldt,  and  John,  1981  p.  1)  An 
attribute  is  defined  as  the  criteria  for  measuring  the  objectives.  Rowe  (1992)  explains  an 
attribute  as  the  quantity  measured  on  an  attribute  scale.  An  example  of  an  attribute  scale 
is  the  dollar  amount  if  the  objective  is  to  minimize  cost.  Figure  2  is  a  simple  example  of 
a  value  tree. 

According  to  Winderfeldt  and  Edwards,  the  theory  of  the  use  of  value  trees  for 
evaluating  options  is  straightforward:  “the  analyst  simply  uses  the  operational  attributes, 
obtains  estimates  of  how  the  options  perform  on  these  attributes,  converts  these  estimates 
to  utilities,  weights  the  attributes,  and  carries  out  the  appropriate  calculations  to  generate 
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an  overall  evaluation  of  the  options.”  (Winderfeldt  and  Edwards,  1986,  p.  45)  However, 
in  practice  this  approach  has  difficulties: 

•  There  may  be  too  many  attributes  to  carry  out  an  evaluation. 

•  Some  branches  of  the  value  tree  (see  Figure  2)  may  be  irrelevant  because 
some  of  the  attributes  might  not  have  effect  on  the  solution  alternatives. 

•  The  decision-maker  may  find  higher  level  branches  of  the  tree  to  be  more 
meaningful  than  the  attributes  itself.  The  task  of  stracturing  relevant  and 
meaningful  operational  attributes  can  be  difficult. 

Despite  the  difficulties  mentioned  above,  a  value  tree  is  a  useful  tool.  Winderfeldt  and 

Edwards  suggest  that,  “value  trees  ^e  most  often  used  as  background  material  in  the 

development  of  attribute  sets  for  a  formal  evaluation.”  (Winderfeldt  and  Edwards,  1986, 

p.45) 


Figure  2.2.  Value  Tree  Example. 
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a.  Purpose  of  Using  Value  Trees 

Winderfeldt  and  Edwards  (1986)  explain  three  purposes  for  using  value 
trees.  First,  value  trees  facilitate  the  process  of  ensuring  that  the  final  set  of  attributes 
actually  captures  all  the  relevant  and  useful  values.  Second,  value  trees  identify  the 
appropriate  level  of  abstraction  at  which  it  is  useful  to  carry  out  the  evaluation.  And 
third,  value  trees  communicate  concisely  the  entire  spectrum  of  values  and  concerns  of 
the  decision-maker  about  the  issue  at  hand. 

b.  Criteria  for  Evaluating  Value  Trees 

Keeney  and  Raiffa  (1976)  propose  the  following  criteria  for  evaluating  the 
objectives  and  the  attributes  in  a  value  tree:  completeness,  operationality, 
decomposability,  absence  of  redundancy,  and  minimum  size.  “Completeness  requires 
that  all  relevant  values  be  included  in  the  superstmcture  of  the  tree,  and  the  substructure 
completely  defines  the  higher  level  values.  Operationality  requires  that  the  lowest  level 
values  or  attributes  be  meaningful  and  assessable.  Decomposability  means  that  the 
attributes  can  be  analyzed  one  or  two  at  a  time,  that  is,  that  they  are  judgmentally 
independent.  Absence  of  redundancy  means  that  no  two  attributes  or  values  mean  the 
same  thing.  Minimum  size  requirement  refers  to  the  necessity  of  keeping  the  number  of 
attributes  small  enough  to  manage.”  (Winderfeldt  and  Edwards,  1986,  p.  43)  However, 
the  authors  have  found  that  these  requirements  do  not  always  work  together.  The  authors 
say,  “These  requirements  conflict.  Operationality  often  requires  further  decomposition, 
thus  increasing  the  number  of  attributes.  Completeness  may  lead  to  redimdancy,  since 
true  value  independence  is  often  an  unattainable  ideal.”  (Winderfeldt  and  Edwards,  1986, 
P-43) 
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3.  Analytical  Hierarchy  Process  (AHP) 

In  the  1970s,  Thomas  L.  Saaty  introduced  an  alternate  method  of  solving  multiple 
criteria  problems  called  Analytic  Hierarchy  Process  (AHP).  AHP  helps  to  solve  the  same 
problems  for  which  MAUT  is  used:  setting  priorities,  generating  a  set  of  alternatives, 
making  decisions  using  benefits  and  costs,  predicting  outcomes  (including  risk 
assessment),  measuring  performance,  ensuring  system  stability,  and  conflict  resolution. 

Thomas  Saaty  and  Luis  Vargas  report  that  in  decision-making,  people  provide 
subjective  judgments  based  on  feelings  and  intuition  rather  than  on  well-formulated 
logical  reasoning  (1991).  “AHP  should  be  used  for  problems  involving  a  large  number  of 
attributes  whose  outcomes  can  only  be  measured  on  a  relative  scale.”  (Tiley,  1994,  p.  27) 
The  Analytical  Hierarchy  Process  (AHP)  demonstrates  a  representation  of  the  mental 
process,  by  which  overall  judgments  come  about  in  situations  involving  complex  goals 
and  criteria.  In  “Prediction,  Projection  and  Forecasting,”  (1991)  Saaty  and  Vargas  define 
AHP  as  a  multi-objective,  multi-criteria  decision-making  approach,  which  employs  a 
pair-wise  comparison  procedure  to  arrive  at  a  scale  of  preferences  among  sets  of 
alternatives.  The  authors  add  that  to  apply  the  AHP  approach,  it  is  necessary  to  break 
down  a  complex  unstructured  problem  into  its  component  parts.  The  parts,  or  variables, 
are  arranged  into  a  hierarchic  order;  assign  numerical  values  to  subjective  judgments  on 
the  relative  importance  of  each  variable;  and  synthesize  the  judgments  to  determine 
which  variables  have  the  highest  priority  and  should  be  acted  upon  to  influence  the 
outcome. 

Saaty  and  Vargas  continue  by  stating  AHP  is  “intimately  connected  with  the  idea 
of  consistency  of  thought.  It  [the  process]  admits  inconsistency  (including  lack  of 
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transitivity)  and  measures  the  effect  of  different  levels  of  consistency  on  the  results” 
(1991,  p.  12)  that  are  sought.  Also,  AHP  “avoids  aprioristic  assumptions  and  models, 
and  calls  on  the  individual  seeking  to  know,  to  construct  the  framework  of  that  quest.” 
(Saaty  and  Vargas,  1991,  p.  12)  The  assumption  in  this  thesis  is  that,  judgment  is  based 
on  the  acquired  experience,  reason,  and  intuition.  The  objective  of  AHP  is  to  “fulfill  the 
purposes  of  the  people  concerned  as  well  as  possible,  rather  than  to  legislate  to  them  that 
the  outcome  should  be  based  on  principles  set  for  by  other  people,  who  are  essentially 
outsiders  to  the  problem.  Thus,  perceived  constraints  must  be  examined  and  not  taken  for 
granted  -  the  only  hope  there  is  to  plan  a  way  out  of  difficult  problems.”  (Saaty  and 
Vargas,  1991,  p.  12) 

The  AHP  synthesizes  two  approaches  by  identifying  the  outcomes  that  are  more 
beneficial  to  the  actors,  and  at  the  same  time,  by  providing  a  way  of  assessing  the  factors 
(causes)  which  may  have  more  to  do  with  certain  types  of  outcomes. 

In  using  AHP,  objectives  must  be  determined  first:  what  needs  to  be  done? 
Second,  generate  alternatives  to  satisfy  those  requirements:  what  are  the  possible  ways  of 
action?  Third,  set  priorities  according  to  the  importance  of  the  requirements  that  allow 
the  implementation  of  the  alternatives  to  attain  some  higher  objective.  Finally,  choose 
the  best  policy  alternative,  or  as  in  some  cases,  a  mix  of  the  best  policy  alternatives.  The 
alternatives  depend  on  the  overall  objective  at  hand.  The  choices  may  be  either  to 
allocate  resources,  assess  the  risk  benefits  and  costs  of  the  allocation,  predict  future 
consequences;  or,  measure  the  level  of  performance  of  the  system,  and  possibly  redesign 
the  overall  structure  to  fit  the  functions  that  must  be  performed  to  meet  certain 
requirements.  If  the  desire  is  to  change  the  structure  of  a  system,  specific  goals  should  be 
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achieved  by  implementation  of  specific  policies.  AHOP  can  assist  in  structuring  problems 
in  a  system,  thereby  assisting  to  prioritize  the  decision-maker’s  objectives. 

AHP  also  has  its  pitfalls,  such  as  rank  reversal.  Rank  reversal  refers  to  the 
concept  of  the  alternative  rankings  that  changes  when  new  information  is  introduced  into 
the  problem  (1994).  The  mathematical  algorithms  used  in  AHP  may  also  be  difficult  to 
explain  to  those  who  are  not  mathematically  inclined.  In  this  thesis  research,  the  need  for 
legitimation  precludes  the  use  of  AHP  in  favor  of  MAUT. 

D.  SOLICITATION  AND  CONSENSUS  BUILDING 
1.  Delphi  Method 

a.  Background 

For  the  purposes  of  this  research,  there  is  a  need  not  only  to  elicit  public 

values,  but  also  to  have  a  kind  of  consensus,  a  method  to  siunmarize  the  elicited  values  of 

several  individuals.  The  Delphi  technique  is  a  method  to  meet  these  needs.  Linstone  and 

Turoff  define  the  Delphi  technique  as  “a  method  for  structuring  a  group  communication 

process  so  that  the  process  is  effective  in  allowing  a  group  of  individuals,  as  a  whole,  to 

deal  with  a  complex  problem.”  (1975,  p.  3) 

The  Delphi  technique  was  developed  during  the  1950s  by  two  researchers 
at  the  RAND  Corporation  as  a  procedure  to  ‘obtain  the  most  reliable 
consensus  of  opinion  of  a  group  of  experts  ...by  a  series  of  intensive 
questionnaires  with  controlled  opinion  feedback.’  (Dalkey  &  Helmer, 

1963)  Initially,  it  [the  Delphi]  was  used  in  the  realm  of  long-term 
forecasts  of  change,  particularly  in  science  and  technology.  (Rowe,  1992) 

In  later  years,  many  researchers  have  employed  the  Delphi  technique  in  a  variety 

of  fields  including  determining  the  educational  policies,  predicting  the  future 
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technological  problems,  effects  on  society,  and  determining  the  future  issues  of  real 
estate. 

According  to  Rowe  (1992),  Delphi  has  four  necessary  characteristics; 
anonymity,  iteration,  controlled  feedback  and  statistical  aggregation  of  group  response. 
Anonymity  is  achieved  through  the  use  of  questionnaires  which  allow  participants  to 
respond  privately.  Anonymity  reduces  the  influence  of  dominant  individuals.  Iteration 
of  is  the  process  of  presenting  the  questionnaire  over  a  number  of  rounds,  which 
encourages  respondents  to  change  their  minds.  Controlled  feedback  takes  place  between 
rounds,  in  which  each  respondent  is  presented  with  the  opinions  of  other  respondents  in 
the  form  of  a  statistical  summary.  At  the  end  of  the  Delphi  procedure,  a  statistical  group 
response  is  obtained  that  illustrates  the  spread  of  opinion  reflecting  the  degree  of 
consensus  on  the  issue. 

b.  The  Delphi  Process 

The  Delphi  process  involves  seven  distinct  tasks  as  shown  in  Figure  3. 
Creating  the  panels  is  the  first  critical  task.  Linstone  and  Turoff  (1975)  consider  three 
kinds  of  panelists  as  ingredients  for  creating  a  successful  mix:  stakeholders;  those  who 
are  or  will  be  affected,  experts,  those  who  have  an  applicable  specialty  or  relevant 
experience;  and  facilitators,  those  who  have  skills  in  clarifying,  organizing,  synthesizing, 
stimulating  individuals  who  can  supply  alternative  global  views  of  the  culture  or  society. 
Judging  the  desirability  of  individuals  considered  for  the  respondent  group  is  subjective 
and  therefore,  according  to  some  critics,  is  subject  to  mistake.  However,  subjectivity  is  a 
problem  when  formulating  any  group  activity-  committees,  study  groups,  etc.  The 
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problem  is  not  imique  to  Delphi,  and  does  not  detract  from  the  method  usefulness. 
(Linstone  and  Turoff,  1975) 


Figure  2.3.  Delphi  method  (Kleindorfer,  Kunreuther  and  Schoemaker,  1993  p.  234). 

A  questionnaire,  prepared  and  distributed  to  participants,  asks  participants 
to  provide  their  opinions  that  they  see  as  being  important.  In  this  research,  the 
respondents  are  asked  to  list  the  most  important  issues  regarding  the  technology  choice 
problem.  “After  the  responses  from  the  members  are  collected  and  all  the  redundant  data 
is  eliminated,  the  responses  are  then  summarized  statistically.  At  the  end  of  round  one,  a 
second-roimd  questioimaire  is  organized,  which  is  entirely  similar  to  the  first-round 


questionnaire,  except  that  a  statistical  summary  of  first-round  responses  from  the  panel  is 
also  enclosed.”  (Kleindorfer,  Kunreuther  and  Schoemaker,  1993,  p.  236) 


The  second  roimd  questionnaire  usually  includes  instructions,  a  statistical 
summary  of  first-round  responses,  and  a  section  where  the  participants  can  provide  any 
additional  comments.  Each  summary  statement  has  a  corresponding  scale  so  that  each 
participant  will  rate  all  the  statements. 

Although  this  research  uses  an  ordinal  scale  to  rate  statements,  many 
Delphi  research  efforts  make  use  of  the  Likert  Scale,  which  is  described  below.  The 
scale  can  have  as  the  lower  anchor,  strongly  disagree  (SD)  -2;  and  the  higher  anchor, 
strongly  agree  (SA)  +2.  Neutral  (N)  is  assigned  a  zero  value,  disagree  (D)  and  agree  (A), 
-1  and  +1,  respectively.  The  zero  point  provides  a  neutral  option  to  panel  members  and  is 
intended  to  overcome  forced-choice  responding  and  provide  members  every  possible 
option  in  rating.  A  threshold  value  can  be  set  at  the  end  such  as  statements  rated  below  +1 
rated  statements  will  not  be  included  in  the  next  round.  The  purpose  of  the  threshold 
value  is  to  eliminate  the  extreme  ideas.  The  responses  of  the  participants  at  the  end  of 
this  round  are  analyzed  by  utilizing  the  descriptive  statistics.  (Clayton,  1997) 

In  the  third  round,  participants  are  provided  the  summarized  data  that  is 
obtained  at  the  end  of  round  two.  Group  mean  and  individual  ratings  are  provided  to  the 
participants  so  that  each  participant  can  see  where  his  or  her  responses  fall  with  respect  to 
the  measure  of  central  tendency.  Each  member  is  then  asked  to  re-rate  the  statements 
having  the  knowledge  of  opinions  expressed  in  the  previous  round.  Participants  are  also 
given  the  opportunity  to  provide  their  reasons  to  their  ratings.  The  responses  are 
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summarized  again,  and  the  participants  are  asked  to  re-rate  the  statements  in  the  light  of 
the  new  summary. 


This  process  goes  on  until  either  a  desired  level  of  consensus  is  reached,  or 
until  a  time  constraint  or  other  constraint  is  reached.  Different  authors  agree  that  a 
successful  Delphi  can  be  carried  out  within  a  three-round  format.  “In  practice,  most 
Delphis  on  policy  try  to  maintain  a  three-  or  four-  roimd  limit  by  utilizing  the  following 
procedure:  (1)  the  monitor  team  devoting  a  considerable  amount  of  time  to  carefully 
preformulating  the  obvious  issues;  (2)  seeding  the  list  with  an  initial  range  of  options  but 
allowing  for  the  respondents  to  add  to  the  lists;  (3)  asking  for  positions  on  an  item  and 
underlying  assumptions  in  the  first  round.”  (Tinstone  and  Turoff,  1975,  p.88)  Usually, 
the  Delphi  process  stops  when  most  of  the  responses  indicate  a  consensus,  or  little  change 
with  further  rounds  is  expected  in  the  distribution  of  responses  from  participants.  This 
research  will  conduct  two  rounds  because  of  time  constraints. 

The  above  described  Delphi  methodology  is  called  conventional  Delphi. 
There  are  two  other  types  of  Delphi  methods,  real  and  policy.  “Real-time  Delphi  differs 
from  conventional  Delphi  in  that,  rather  than  taking  weeks  to  conduct  the  process,  it 
occurs  during  the  course  of  a  meeting  or  conference.”  (Clayton,  1997,  p.  377)  In  the  real¬ 
time  Delphi,  the  monitor  team  is  replaced  by  a  computer,  which  has  been  programmed  to 
cany  out  the  compilation  of  the  group  results.  This  approach  has  the  advantage  of 
eliminating  the  delay  caused  in  summarizing  each  round  of  Delphi,  thereby  turning  the 
process  into  a  real-time  communication  system.”  (Tinstone  and  Turoff,  1975,  p.5) 

The  Policy  Delphi  is  a  technique  used  to  achieve  “every  possible  opposing 
view”  (Tinstone  and  Turoff,  1975,  p.84)  on  an  issue.  According  to  Clayton  (1997),  in 
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Policy  Delphi,  the  decision-maker  is  not  interested  in  having  a  group  generate  his 
decision,  but  rather  in  having  and  informed  group  present  all  the  options  and  supporting 
evidence  for  his/her  consideration.  The  policy  Delphi  is  not  a  mechanism  for  making 
decisions  as  generating  a  consensus  is  not  the  prime  objective.  “A  policy  Delphi  should 
be  able  to  serve  one  or  any  combination  of  the  following  objectives:  (1)  to  ensure  that  all 
possible  options  have  been  put  on  the  table  for  consideration;  (2)  to  estimate  the  effect  of 
any  particular  option;  (3)  to  examine  and  estimate  the  acceptability  of  each  option.” 
(Tinstone  and  Turoff,  1975,  p.  84) 

The  major  strengths  of  Delphi  are  summarized  as:  Delphi  technique  helps 
to  establish  convergence  of  ideas  over  an  issue.  In  Delphi,  questionnaires  can  be  sent  to 
participants  by  mail  or  other  means,  eliminating  the  need  to  travel  long  distances  to  get 
together  at  the  same  place  and  time.  Therefore,  Delphi  is  cost  effective.  During 
implementation  of  Delphi,  participants  are  required  to  provide  their  responses  to  the 
questionnaires  privately.  This  process  eliminates  the  negative  effects  of  dominant 
groups,  rank,  personality,  etc. 

Tinstone  and  Turoff  (1975)  identified  inherent  difficulties  in  the  Delphi 
method.  These  include  technical  difficulties  of  re-adapting  the  method  for  each  issue,  the 
arbitrariness  of  panel  members  (respondents)  selection,  the  balancing  of  closed  and  open- 
ended  responses  in  the  questionnaire,  and  estimating  the  time  required  for  completion. 
At  the  end  of  each  round,  controlled  feedback  is  provided  to  the  respondents.  In  the 
process  of  feedback,  limiting  or  eliminating  some  of  the  alternatives  might  have  negative 
effects  on  the  success  of  Delphi. 
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The  number  of  respondents  depends  on  the  type,  purpose  and  the 
complexity  of  the  subject.  According  to  Tinstone  and  Turoff  (1975),  real-time  Delphi 
and  conventional  Delphi  can  be  given  from  a  small  to  a  large  group  and  a  policy  Delphi 
can  be  given  to  a  group  ranging  from  ten  to  fifty  people.  This  research  is  categorized  as 
Policy  Delphi,  but  utilizes  fewer  respondents  than  Tinstone  and  Turoff  suggested. 

E.  AGGREGATION  METHODS 

Aggregation  is  used  to  measure  the  level  of  consensus  derived  from  the  Delphi 
method.  Aggregation  is  defined  as  collecting  inputs  into  a  mass,  yielding  a  collective 
output.  It  has  been  studied  in  mathematics,  and  has  expanded  into  the  decision  sciences. 
When  individual  opinions  are  aggregated,  the  result  is  a  single  opinion,  representing  a 
group  choice.  Examples  of  basic  aggregation  methods  include,  mean,  median,  mode  and 
Nominal  Group  Technique.  Advanced  methods  presented  in  this  section  include  the 
Hungarian  method  and  Median  Ranking  method. 

1.  Basic  Methods  of  Aggregation 

Consider  the  following  example  of  several  Restoration  Advisory  Board  (RAB) 
members  listing  various  attributes  in  a  Delphi  forum.  The  panelists  are  comprised  of 
seven  different  RAB  members,  each  from  a  different  location.  The  panelists  define  the 
attributes  during  the  Delphi  forum,  and  rank  them  in  order  of  importance  according  to 
their  respective  RABs.  No  ties  among  the  attributes  are  allowed  (see  Figure  4). 
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Mean,  also  known  as  average,  is  the  most  common  measure  of  location  for  a  set 
of  numbers.  According  to  Wheeler  and  Chambers,  average  identifies  the  center  of  mass 
for  the  values  in  the  data  set  (1992).  Therefore,  it  can  be  considered  the  balance  point 
among  the  data.  The  calculation  of  mean  is  fairly  simple.  It  is  the  sum  of  all  the  values 
(In  this  example,  the  number  represents  the  ordinal  ranking  of  attributes,  showing  the 
relative  importance  of  each  attribute,  i.e.  5  means  5*  most  important,  3  means  most 
important,  etc.)  divided  by  the  total  number  of  values.  Consider  the  above  example:  the 
total  score  of  Attribute  1  is  28  (the  sum  of  5,  2,  2,  4,  1,  6,  7).  The  mean  is  the  sum  of 
these  seven  values,  divided  by  7  (28/7  =  4).  This  value  is  the  balance  point  for  Attribute 
1,  and  the  mean  values  for  Attributes  2  through  7  are  presented  in  Figure  6.  Therefore,  if 
the  RAB  members  decided  the  lowest  number  was  the  most  desired,  then  Attribute  5 
would  be  the  best  decision. 
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Figure  2.4.  Example  RAB  Attribute  Table. 
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A  median  is  another  measure  of  location  for  a  data  set.  This  statistic  identifies  the 
50'*’  percentile  of  the  data,  indicating  at  least  half  the  data  are  greater  than,  or  equal  to  the 
median;  and  at  least  half  the  data  are  less  than,  or  equal  to  the  median.  Therefore,  while 
mean  describes  the  balance  point  of  the  data  set,  median  splits  the  data  set  into  two  equal 
halves.  In  calculating  the  median,  the  first  step  is  to  arrange  the  data  set  into  numerical 
order  (see  Figure  5). 

The  revised  example  would  be: 
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Figure  2.5.  Defining  the  Term  Median  in  the  Example  RAB  Attribute  Table. 

Next,  find  the  middle  of  this  ordered  data  set.  If  the  data  set  contains  an  odd 
number  of  values,  there  will  be  one  middle  value;  if  the  set  is  even,  there  will  be  two 
middle  values,  and  the  median  is  calculated  by  averaging  these  two  middle  values.  In  the- 
example,  the  data  set  contains  a  total  of  7  values;  therefore,  the  middle  value  for  each 
attribute  is  listed  in  the  4'*’  column  (see  Figure  5).  The  problem  with  this  data  set  is  there 
is  a  4-way  tie  among  the  attributes,  and  the  data  set  provided  does  not  give  an  ordinal 
ranking,  therefore  the  decision-makers  do  not  know  which  attributes  are  ranked  1®*,  2"** 
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and  so  on.  The  median  method  does  not  give  the  optimal  solution;  however,  it  explains 
where  the  RAB  members  have  a  general  tendency  towards  an  alternative,  while  not 
defining  which  alternative  is  optimal. 

Mode  is  the  measurement  that  occurs  most  frequently  in  the  data  set.  Referring  to 
Figure  6,  the  mode  is  identified  for  each  attribute.  Attributes  1,  2,  and  3  do  not  have  a 
mode;  therefore,  the  decision-maker  cannot  rank  this  data  set  to  derive  the  preferred 
choice  among  the  RAB  members. 

When  the  decision-maker  assigns  equal  importance  among  all  the  attributes,  the 
simplest  way  to  obtain  an  overall  ranking  is  to  total  each  row  and  rank  them  in  ascending 
order  (Yoon  and  Hwang,  1995).  This  technique  is  called  the  Nominal  Group  Technique, 
which  provides  an  ordinal  ranking  of  the  alternatives;  yet  has  its  downfalls.  First  of  all,  it 
does  not  consider  conflicting  scores  given  by  the  RAB  members.  For  example,  in  figure 
4,  RAB(A)  ranks  Attribute  4  as  the  best  (score  of  1)  and  RAB(E)  ranks  Attribute  4  as  the 
worst  (score  of  7).  This  technique  is  a  consensus  building  technique,  where  arithmetic 
operations  do  not  provide  the  optimal  solution,  but  presents  a  data  set  that  every  voting 
member  can  support.  Figure  6  is  a  revised  Delphi  table  listing  the  covered  aggregation 
methods: 

2.  Advanced  Methods  of  Aggregation 
a.  Median  Ranking  Method 

Median  Ranking  Method  is  used  to  measure  the  amount  of  agreement  or 
disagreement  among  rankings.  The  method  is  done  by  a  metric  or  distance  function  for 
the  set  of  rankings.  Median  and  consensus  ranking  is  calculated  by  minimizing  the  total 
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absolute  distance.  Using  the  scores  given  in  Figure  4,  the  method  explained  by  Cook  and 
Seiford  (1978)  follows. 

There  are  7  RAB  members  and  7  attributes.  The  formula  that  is  used  to 
derive  a  distance  matrix  is  represented  by 

m 

dik  =  SJai’-k|, 


MEDIAN  TOTAL  MODE 


MEAN  NGT 


Figure  2.6.  Example  RAB  Attribute  Table  with  Listings  of  Mean,  Median,  Mode  and 

NGT. 

where  dik  represents  the  distance  matrix,  ^  defines  the  number  of  rankings  (or,  in  the 
distance  matrix,  the  column  number).  In  this  research,  each  RAB  represents  one  ranking. 
The  symbol  /  represents  the  number  of  members  (in  this  case,  RAB  identification).  The 
symbol  i  represents  the  number  of  attributes  (or,  in  the  distance  matrix,  the  row  number). 
The  distance  matrix  is  the  sum  of  the  deviations  between  a  selected  ranking  (k)  and  the  7 
given  member  rankings  (a,')  of  an  attribute: 
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dll  =  |5-1|  +  |2-1|  +  |3-1|  +  |4-1|  +  |1-1|  +  |6-1|  +  |7-1|  =  21 
d73  =  |6-3|  +  |7-3|  +  |2-3|  +  |1-3|  +  |5-3|  +  |4-3|  +  |6-3|  =  16 
The  complete  calculation  gives: 


DISTANCE  MATRIX  (djk) 
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Figure  1.7.  Distance  Matrix  of  RAB  Ranked  Attributes  Using  Median  Ranking  Method. 

After  calculating  the  distance  matrix,  the  Himgarian  method  can  be  used  to 
minimize  the  total  distance.  The  output  of  the  Himgarian  method  should  give  the  optimal 


ranking. 


b.  Hungarian  Method 

The  output  of  the  Median  Ranking  method  is  the  input  to  the  Hungarian 
method.  The  Hungarian  method  is  used  in  minimization  problems  in  operations  science 
such  as  minimizing  completion  time  of  a  project,  minimizing  total  cost,  or  minimizing 
the  total  distance  among  attributes.  Referring  to  the  distance  matrix  in  figure  7,  the 
following  steps  are  taken: 

•  For  each  row,  subtract  the  minimum  number  in  that  row  fi'om  all  numbers 
in  that  row. 

•  For  each  column,  subtract  the  minimum  number  in  that  colunm  from  all 
numbers  in  that  column. 

•  Draw  the  minimum  number  of  lines  to  cover  all  zeroes.  If  this  number  = 
m,  stop.  An  assignment  can  be  made. 
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Determine  the  minimum  uncovered  number  (call  it  d) 


a.  Subtract  d  from  uncovered  numbers. 

b.  Add  d  numbers  covered  by  two  lines. 

c.  Numbers  covered  by  one  line  remain  the  same. 

d.  Then  go  to  the  previous  step. 

After  completing  the  above  steps,  the  distance  matrix  now  looks  like: 
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Figure  2.8.  Minimization  of  the  Distance  Matrix. 

The  steps  to  determine  the  optimal  solution  are: 

•  Find  a  row  or  column  with  only  one  limited  zero  and  circle  it. 

•  If  the  circle  is  in  a  row  with  one  zero,  draw  a  line  through  its  column.  If 
the  circle  is  in  a  column  with  one  zero,  draw  a  line  through  its  row.  One 
approach,  when  all  rows  and  columns  have  two  or  more  zeroes,  is  to  draw 
a  line  through  one  with  the  most  zeroes,  breaking  ties  arbitrarily. 

•  Repeat  the  previous  until  all  the  circles  are  lined.  If  this  minimum  number 
of  lines  equals  m,  where  m  is  the  number  of  jobs  and  workers,  then  the 
circles  provide  the  optimal  assignment 
(htpp://isgs.ba.ttu.edu/classes/jbums/Tsq5342/slides/ch07). 


The  final  outcome  of  the  assignment  problem  using  the  Hungarian  method  is: 
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Figure  2.9.  Assignment  Matrix. 


3.  Assigning  Weights  to  the  Aggregated  Ranking 

Once  the  final  ranking  is  determined  through  the  Hungarian  method,  the  decision¬ 
maker  needs  to  compute  the  weights  for  the  attributes  with  respect  to  one  another.  In 
other  words,  how  much  more  important  is  the  highest-ranked  attribute  to  the  second 
highest-ranked  attribute?  The  “Smarter”  method,  used  in  a  software  program  called 
Logical  Decisions®  for  Windows'^^  (LDW),  uses  the  final  ranking  from  the  aggregation 
method  as  its  input,  and  a  least  preferred  and  most  preferred  level  is  given  by  the 
decision-maker  to  compute  the  weights  for  each  attribute  accordingly.  For  example, 
there  are  three  attributes,  project  time,  completeness  and  total  cost,  ranked  as  first  and  a 
tie  for  second  respectively  (see  Figure  10). 


Attribute 

Rank 

Least  Preferred 
Level 

Most  Preferred 
Level 

Project  time 

1 

4  years 

1  year 

Completeness 

2 

50% 

100% 

Total  Cost 

r  2 

$10  Million 

$1  Million 

Figure  2.10.  Example  of  Assessment  Using  the  “Smarter”  Method. 
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The  “Smarter”  method  will  take  the  ranks  and  assign  a  higher  weight  to  project 
time,  and  then,  lower  weights  to  completeness  and  total  cost.  For  each  choice  sediment 
technology,  there  will  be  a  level  for  each  attribute  listed  in  Figure  10.  For  example, 
sediment  technology  may  take  3  years  to  be  80%  complete,  and  a  total  cost  of  $7  million. 
The  total  utility  score  is  calculated  by  the  “Smarter”  method  with  these  figures  and  the 
rank  of  each  attribute.  After  all  utility  scores  are  calculated  for  each  sediment 
technology,  the  technology  with  the  highest  utility  score  is  the  optimal  decision. 
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III.  DESIGN  IMPLEMENTATION  AND  RESULTS 


A.  INTRODUCTION 

This  chapter  outlines  the  methodology  used  to  implement  Value  Tree  Analysis 
through  the  use  of  the  Delphi  Method  conducted  via  the  internet.  The  first  four  sections 
describe  the  research  of  applications  used  to  create,  manage  and  publish  a  website  and 
survey,  and  use  of  a  file  transfer  service  to  upload  and  download  files  to  web  servers. 
The  fifth  section  discusses  the  formulation  of  roles  of  proxy-Restoration  Advisory  Board 
(RAB)  members.  The  sixth  section  explains  the  implementation  of  the  Delphi  method 
via  the  internet-based  discussion  forum  and  survey.  The  process  and  results  of  the 
discussion  phase  and  the  two  voting  rounds  are  detailed.  The  last  section  provides  the 
overall  results  of  the  survey  experiment. 

B.  WEBSITE  DESIGN 

1.  Requirements 

RABs  need  a  common  medium  that  provides  access  to  sediment  technology 
treatment  information  to  educate  themselves  as  well  as  others  on  important  public  issues, 
and  to  commimicate  their  concerns  and  desires  to  public  decision-makers.  The 
instruction  and  the  methodology  information  provided  on  the  web  provides  another  way 
to  implement  Delphi,  which  typically  uses  methods  including  conference  meeting  and 
mail  out  services  in  the  first  phase  of  execution.  An  electronic-based  discussion  forum 
provides  proxy-RABs  with  the  ability  to  post  messages  about  their  concerns,  ask 
questions,  read  others’  messages,  reply  to  any  posted  message,  and  chat  if  they  wish  to  do 
so.  An  advantage  using  internet  and  web  technology  is  to  allow  the  proxy-Restoration 
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Advisory  Board  (RAB)  members  to  meet  virtually  to  discuss  and  to  post  their  comments, 
questions  and  responses  at  their  convenience. 

2.  Website  Design  Tools  and  Free  Web  Server  Providers 

The  authors  of  this  research  considered  using  several  different  website  design 
tools  to  facilitate  the  Value  Tree  Analysis  and  the  Delphi  process.  A  brief  discussion  of 
the  products  considered  is  provided.  Ultimately,  the  researchers  selected  Netscape® 
Composer,  WS_FTP95,  Xoom.com  and  Delphi.com  for  the  research.  Reasons  for  the 
choices  are  provided. 

Microsoft®  Frontpage®  98  is  a  tool  to  create  the  web  site.  “Microsoft® 
Frontpage®  98  consists  of  the  FrontPage®  Explorer  (a  site  management  tool),  the 
Frontpage®  Editor  (an  HTML  processor),  image  composer  (with  which  you  can  create 
and  edit  both  still  and  animated  graphics),  and  three  web  server  programs  (from  which 
you  choose  depending  on  your  platforms)”  (Tauber  and  Kienen,  1998).  Microsoft® 
Frontpage®  98  has  the  following  capabilities; 

•  Creates  and  edits  HTML  easily,  previews  the  page  created  and  views  the 
HTML  source  code 

•  Designs  tables,  resizes  rows  and  columns  quickly 

•  Includes  forms  where  the  data  can  be  routed  via  e-mail  or  saved  to  a  server 

•  Includes  Microsoft®  Office  documents  in  HTML  pages 

•  Includes  text  animation  and  other  special  effects 

•  Assigns  permission  to  view  and  change  various  parts  of  the  designer’s  site 

Netscape®  Composer  is  another  tool  used  to  create  or  edit  web  pages  containing 

images,  tables  and  links;  and  publishes  the  final  drafted  website  on  the  web.  Netscape® 
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Composer  is  easy  to  use,  and  its  simplicity  is  similar  to  using  a  word  processor.  The  files 
that  the  composer  creates  are  HTML  (Hypertext  Markup  Language)  files  just  as 
Microsoft©  Frontpage®  creates.  When  the  web  pages  (HTML  files)  are  published  on 
the  web,  the  browsers  know  how  to  interpret  the  HTML  codes  to  display  HTML  files. 

Most  of  the  Internet  Server  Providers  (ISPs)  include  HTML  editing  tools  to  create 
a  personal  page.  If  the  user  is  not  a  member  of  an  ISP  (such  as  America  Online  ©  1999), 
there  are  many  Free  Web  Service  Providers  (FWPs)  available  on  the  web  to  provide  the 
necessary  tools  to  create  and  host  web  sites  for  fi'ee. 

FWP  services  are  good  if  the  number  of  pages  created  do  not  exceed  fifty.  Most 
FWP  providers  have  limitations  on  certain  web  page  features,  such  as  computer  hard 
drive  space  (see  Table  3.1).  Therefore,  if  the  user  desires  to  have  a  separate  domain 
name,  create  hundreds  of  pages,  use  different  web  technologies  (such  as  secure 
transactions),  and  expect  high  traffic  capacity;  then  FWP  cannot  be  used.  For  further 
information  regarding  free  web  providers,  refer  to  http://netguide.com.  FWP  providers 
usually  provide  services  including  firee  e-mail,  file  management  and  File  Transfer 
Protocol  (FTP).  FWP  providers  usually  generate  revenues  through  advertising. 


Free  Web  Service  Provider 

Hard  Drive  Space  (MB) 

Xoom 

11 

Crosswinds 

Unlimited 

AcmeCity 

20 

Angelfire 

5 

Hypermart 

10 

Table  3.1.  Examples  of  FWPs  with  Available  Hard  Drive  Capacities. 
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Delphi  Forums  Inc.,  1999,  provides  a  free  elecfronic  mediiim  for  open  discussion 
that  allows  members  to  create  and  manage  personalized  online  forums.  Delphi  Forums 
provides  a  medium  to  connect  and  exchange  ideas  through  a  variety  of  fiill-featured,  web- 
based  messaging  services.  Visitors  have  the  capability  to  browse  through  messages  left 
by  others,  respond  to  previous  postings,  or  start  new  conversations.  Delphi  Forums 
includes  real  chat  and  file  attachment  features.  In  addition,  the  designer  of  the  forum  has 
the  ability  to  limit  the  access  by  designating  the  forum  as  a  private  website  vice  a  public 
website.  For  more  information  regarding  the  use  of  Delphi  Forums,  visit  its  website  at 
www.delphi.com. 

This  thesis  project  uses  Delphi  Forums  because  its  services  meet  the  requirements 
of  this  research.  Delphi  also  provides  tools  to  build  a  site  homepage,  which  offers  a  link 
to  the  discussion  forum.  A  limitation  of  Delphi  Forums  is  the  inability  to  allow  more 
than  one  web  page  per  website;  therefore,  another  website  provider  is  used  to  host  links 
to  the  Delphi  Forums  homepage.  The  online  forum  is  designed  as  a  private  website  to 
allow  only  invited  proxy-RAB  members  access  to  the  forum.  Proxy-RAB  members  are 
assigned  unique  usernames  and  passwords  by  the  forum  hosts. 

On  the  advice  of  fellow  Information  Technology  students,  Xoom.com,  Inc.  1997 
is  used  for  hosting  the  website  created  in  this  thesis  research.  The  only  requirement  while 
registering  with  Xoom.com  is  to  provide  personal  information  including  user  name,  e- 
mail  address  and  affiliated  company.  To  create  the  link  pages,  Netscape®  Composer  is 
used.  After  creating  the  desired  web  pages,  the  web  designer  publishes  the  web  pages  on 
a  web  server  (Xoom.com). 
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This  thesis  research  uses  WS_FTP95  to  upload  the  designed  web  pages  (HTML 
pages)  to  the  web  server.  To  upload  the  web  pages  to  the  web  server,  the  web  designer 
must  have  access  to  the  web  server.  During  the  uploading  process,  the  FTP  program  asks 
the  web  designer  to  enter  a  profile  name,  host  name,  user  name  and  password  to  be  able 
to  connect  to  the  server  and  perform  the  necessary  file  transfers. 

3.  File  Transfer  Protocol  (FTP) 

“The  most  widely-deployed  Internet  file  transfer  service  uses  the  FTP.  FTP 
permits  transfer  of  an  arbitrary  file,  and  includes  a  mechanism  that  allows  files  to  have 
ownership  and  access  restriction.  A  file  transfer  service  can  move  a  copy  of  a  file  from 
one  computer  to  another,  either  interactively  or  in  batch  mode.”  (Comer,  1997,  p.  347, 
348) 

FTP  is  based  on  the  client-server  model  of  communication  between  a  server  and  a 
client  computer.  The  server  provides  services  such  as  transferring  the  requested 
information  to  the  client  computers.  The  client  computers  request  information  from  the 
server.  Non-government  and  government  institutions  usually  have  an  FTP  server  to  post 
their  materials  to  the  interested  parties  or  individuals.  In  this  thesis,  an  application  of 
FTP  is  used  to  upload  the  HTML  pages  that  are  created. 

C.  SURVEY  TOOLS 

This  thesis  research  focuses  on  the  subjective  input  of  the  stakeholders  who  are 
represented  by  proxy-RAB  members.  The  importance  of  eliciting  the  concerns  of 
members  regarding  the  San  Francisco  Bay  Sediment  Project,  and  then  allowing  the  group 
to  rank  the  overall  set  of  concerns  ordinally,  is  required  to  utilize  public  opinion  in  the 
decision-making  process.  Currently,  public  representatives  (RAB  members)  are  asked  to 
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attend  local  meetings,  where  discussion  forums  take  place  to  identify  the  issues  pertinent 
to  the  problem.  The  goals  of  utilizing  a  web  survey  are  to  eliminate  the  need  for  a 
physical  meeting  by  conducting  on-line  discussions,  and  by  eliciting  issues  over  the 
internet. 

1.  Research  Requirements 

The  survey  tool  must  fulfill  several  functions.  First  the  survey  must  provide  an 
internet-based  platform  to  perform  the  survey.  Second,  the  tool  must  allow  a  large 
number  of  inputs  by  voting  members.  Third,  the  tool  must  allow  members  to  rank  the  set 
of  concerns  (changed  into  attributes)  in  an  ordinal  manner.  Fourth,  the  tool  must  provide 
the  individual  results  of  the  survey  to  the  survey  host  via  email  in  text  or  database  format, 
and  finally,  the  tool  must  provide  the  entire  service  free  of  charge. 

2.  Alternative  Survey  Tools 

The  internet  provides  a  host  of  websites  that  allow  the  users  to  develop  surveys 
free  of  charge.  However,  the  free  survey  providers  have  limited  capabilities  in 
questionnaire  design,  storage  and  publishing  results.  The  following  free  internet 
providers  and  its  capabilities  are  explained. 

Delphi  Forums,  Inc.  1999  allows  stakeholders  to  voice  their  opinions 
anonymously  via  internet  discussion  forum,  the  ability  to  rank  a  set  of  attributes  is 
unavailable.  Instead,  Delphi  Forums  allows  the  fonun  host  to  list  up  to  five  choices, 
where  the  voting  member  may  choose  the  most  important  choice  alternative.  In  other 
words,  the  voting  allowed  in  Delphi  Forums  allows  the  stakeholder  to  cast  a  vote  in  a 
ballot-type  format.  For  more  information  regarding  the  use  of  Delphi  Forums,  visit  the 
website  at  www.Delphi.com. 
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Perseus  SurveySolutions^“  for  the  Web  is  an  application  for  creating  and 
distributing  surveys,  as  "well  as  collecting,  analyzing  and  reporting  results.  While  some  of 
the  most  common  applications  used  by  Perseus.com  include  customer  and  employer 
satisfaction  surveys,  course  evaluations,  and  opinion  polls,  Perseus  SurveySolutions'^“_/br 
the  Web  also  creates  templates  for  lead  generation  forms,  technical  support  inquiry  forms, 
web-feedback  questionnaires,  and  order  forms.  One  of  the  most  impressive  features  of 
Perseus  SurveySolutions"^”  for  the  Web  is  its  ability  to  produce  instant  survey  results, 
making  Perseus  SurveySolutions"^”  for  the  Web  ideal  for  publishing  entertaining  and 
informative  polls  on  a  personal  web  site.  For  more  information  regarding  survey 
development  with  Perseus  SurveySolutions™  for  the  Web,  visit  its  website  at 
WWW  .Perseus,  com. 

This  thesis  research  uses  SurveySolutions™  for  the  Web,  developed  by  Perseus 
Development  Corporation.  This  tool  uses  an  intuitive  word-processing  interface  for 
survey  design,  and  automates  HTML  translation  for  web-siuweys  and  e-mail  conversion 
for  surveys  that  will  be  distributed  by  electronic  mail.  Survey  responses  are  collected 
either  on  the  server  or  on  a  workstation.  Results  can  be  stored  in  a  variety  of  formats 
including  standard  ASCII  TSV  (tab  separated  values)  files  on  the  server  and  Microsoft 
access  databases  on  your  workstation.  Perseus  SurveySolutions™  for  the  Web  also 
allows  a  free  30-day  trial  version,  which  allows  this  thesis  to  be  conducted  at  no  cost. 

D.  FINAL  WEBSITE  AND  SURVEY  DESIGN 

Upon  choosing  the  free  applications  for  creating,  managing  and  publishing  a 
website  and  survey,  the  creation  of  the  NPS  Sediment  Project  Home  Page  was  formed 
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( www.delphi.com/nps  survey').  The  research  homepage  and  the  survey  are  shown  in 
Figures  3.1  and  3.2. 


NPS  Sediment  Project  Home  Page 


Background  of  the  Sediment  Project: 

As  a  result  of  implementation  of  Base  Realignment  and  Closure  Commission  (BRAC) 
decisions,  the  naval  operations  in  the  San  Francisco  Bay  area  has  declined  precipitously. 
Yet  a  legacy  of  the  naval  presence  remains;  portions  of  the  Bay  are  contaminated  from 
years  of  naval  operations.  The  San  Francisco  Bay  Sediment  project  is  the  U.S.  Navy's 
program  directed  at  redressing  the  contamination  of  sediment  from  the  Bay  area  naval 
operations  near:  Mare  Island,  Himters  Point  Shipyard,  Alameda  Point  and  Treasure 
Island.  The  project  is  administered  by  Engineering  Facility  Activity  (EFA),  West  which 
is  responsible  for  the  project  design  as  well  as  project  execution.  Two  students  at  the 
Naval  Postgraduate  School  have  designed  a  communication  process  (that  EFA,  West  can 
consider  for  implementation)  that  could  help  EFA,  West  elicit  and  utilize  public  concerns 
regarding  site  cleanup.  The  methodology  is  specifically  aimed  at  assisting  EFA,  West 
officials  in  selecting  the  best  remedy  technology  for  each  of  the  contaminated  sites.  The 
methodology,  designed  by  the  students,  involves  the  public  in  the  technology  choice- 
decision  m^ng  process  and  ensures  that  these  concerns  are  a  part  of  the  decision¬ 
making  for  each  site.  For  more  information,  please  go  to  the  link  entitled  "Methodology 
Information."  Proxy-RAB  members,  those  designated  as  participants  in  the  methodology 
experiment,  are  instructed  to  read  the  "Methodology  Information"  before  reading  the 
"Instruction  for  Proxy-RABs"  link. 


Relevant  links: 

Message  Board 
Chat 

Treatment  Perspectives 
NAVFAC  Homepage 

Navy’s  5-vear  environmental  restoration  pgm 

Sample  Survey 

Methodology  Information 

Instruction  for  Proxv-RABs 

Survev-Phase2 

Sur\-ev-Phasel  Results 


Figure  3.1 .  NPS  Sediment  Project  Home  Page. 


NPS  Pilot  Sediment  Project  Survey 

Please  review  and  rank  the  following  attributes  according  to  your  personal  preference 
(1  =  most  important,  24  =  least  important).  Do  not  assign  any  ties.  Feel  jfree  to  use  the  'other' 
section  to  add  any  attribute  you  feel  is  important,  but  not  included. 

Thank  you  for  your  participation. 

Attributes. 

#  of  domestic  pet  deaths 

#  of  domestic  pet  poisonings 

#  of  civil  lawsuits  filed 

#  of  complaints  or  protests 

#  of  on-site  injuries 

Rate  of  medical  treatments  in  area 

#  of  birth  defects  in  area 
Amount  of  contaminant  disposal 

Distance  of  disposal  site  from  residential  area 

Contaminants  in  air 

Contaminants  in  soil 

Quality  of  drinking  water 

Migration  rate  of  drinking  water 

Availability  for  recreational  boating/fishing 

Total  cost  of  site  cleanup 

Amount  of  total  depreciation  of  land 

#  of  fish/wildlife  deaths 

#  of  fish/wildlife  poisonings 
Consumable  seafood? 

Project  completion  time 

Stability  of  foundation  (after  treatment) 

Time  for  site  to  naturally  restore  itself 
Future  reuse  for  residential  area? 

Future  reuse  for  business  district? 

2.  Please  provide  any  comments  about  your  ranking.  If  you  have  an  additional  attribute 
that  should  be  considered,  list  them  in  the  below  box  and  provide  justification  for  its 
inclusion. 


Figure  3.2.  Survey  used  for  Delphi  Rounds  I  and  II. 
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E.  IMPLEMENTATION  OF  DELPHI  METHOD 

The  general  steps  of  the  Delphi  method  are  outlined  in  the  literature  review 
chapter  (Chapter  2,  Section  D.l.b).  This  section  describes  the  actual  methodology  used 
in  eliciting  stakeholder  value  and  aggregating  the  data  to  give  a  final  ordinal  ranking  of 
the  stakeholders’  set  of  attributes  for  selecting  the  optimal  sediment  technology 
treatment. 

1.  Proxy-Restoration  Advisory  Board  (RAB)  Selection 

The  thesis  researchers  originally  plaimed  to  use  actual  RAB  members  involved  in 
the  San  Francisco  Bay  Sediment  Project;  however,  time  constraints  and  other 
commitments  by  EFA,  West  members  limited  their  participation  at  this  time.  The 
available  information  about  sediment  treatment  perspectives  and  NAVFAC’s  Five  Year 
Restoration  Program)  is  used  to  develop  the  proxy-RAB  roles. 

Proxy-RAB  members  with  personal  computers  and  internet  access  were  chosen. 
Of  the  fourteen  candidates  originally  requested  to  participate  in  the  experiment,  seven 
agreed  to  take  a  role  as  one  of  the  seven  panelists  representing  RABs  from:  Alameda 
Naval  Air  Station,  Concord  Naval  Air  Station,  Mare  Island  Naval  Shipyard,  Moffett  Field 
Naval  Air  Station,  Treasure  Island  Naval  Station,  Treasure  Island  Naval  Station  Hunters 
Point  Annex,  Oakland  Fleet  and  Industrial  Supply  Center.  The  panelists’  demographic 
data  include: 

-  0-5  Navy  Flight  Officer,  stationed  in  Monterey,  CA 

-  0-5  Marine  Corps  Reservist,  stationed  in  Quantico,  VA 

-  0-4  Retired  Limited  Duty  Officer,  residing  in  San  Diego,  CA 
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-  0-3  Naval  Advisor,  stationed  in  Jacksonville,  FL 

-  0-3  Judge  Advocate  General  Officer,  stationed  in  Bahrain 

-  0-3  Fleet  Support  Officer,  stationed  in  Monterey,  CA 

-  0-3  Civil  Engineer  Officer,  station  in  Monterey,  CA 

Each  panelist  was  emailed  information  specific  to  the  site  they  represented.  The 
roles  of  each  RAB  member  are  shown  in  Appendix  B. 

2.  Discussion  Phase  of  Delphi 

As  explained  in  the  Literature  Review  Chapter,  the  goal  of  the  discussion  phase  in 
the  Delphi  is  to  identify  the  public  issues  relevant  to  the  San  Francisco  Bay  Sediment 
Treatment  Project  (Chap.  2,  Sect.  D.l.a).  The  proxy-RAB  panelists  are  given  a  period  of 
one  week  to  review  their  specific  role,  and  to  identify  the  issues  which  are  considered 
important  in  selecting  the  best  sediment  treatment  technology.  The  panelists  are 
instructed  to  go  to  the  NPS  Sediment  Project  Home  Page  (www.delphi.com/nps_survey), 
where  the  panelists  have  access  to  background  information  and  other  relevant  links 
regarding  sediment  restoration  programs.  Instructions  for  participating  in  the  discussion 
phase  of  the  Delphi  are  also  given,  and  panelists  are  directed  to  use  this  electronic 
discussion  forum  to  provide  their  concerns  about  the  sediment  treatment  project.  The 
leading  question  posted  on  the  discussion  forum  is,  “As  a  Restoration  Advisory  Board 
(RAB)  member  involved  in  the  San  Francisco  Bay  Sediment  Project,  what  are  the  most 
important  issues  regarding  sediment  treatment?  (Provide  a  minimum  of  3  issues)”  The 
unedited  responses  of  the  participating  proxy-RAB  panelists  are  shown  in  Appendix  C. 
The  results  of  the  discussion  phase  are  interpreted  into  measurable  attributes,  and 
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constructed  into  a  value  tree  representing  all  the  issues  articulated  by  panelists  relevant  to 
selecting  the  best  treatment  technology. 

Of  the  seven  proxy-RAB  panelists,  only  five  responded  to  the  discussion 
question.  The  five  participants  gave  a  total  of  27  issues,  24  of  which  are  relevant  to 
choosing  the  best  treatment  technology.  Figures  3.3  is  the  constructed  value  tree  and 
Figure  3.4  summarizes  the  attributes  articulated  by  the  proxy-RAB  panelists. 
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Figure  3.3.  Value  Tree  Derived  from  Proxy-RAB. 
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ISSUES  PRESENTED  BY  PROXY-RAB  PANELISTS 

#  of  domestic  pet  deaths:  How  many  pets  have  died  as  a  result  of  exposure  to  the 
sediment  treatment  technology? 

#  of  domestic  pet  poisonings:  How  many  local  pets  can  be  infected  with  a  disease  if 
exposed  to  the  sediment  treatment  technology? 

#  of  civil  lawsuits:  How  many  class  action  lawsuits  have  been  filed  as  a  result  of 
exposure  to  the  combination  of  contaminants  and  sediment  treatment  technology? 

#  of  complaints  or  protests:  How  many  complaints  have  been  filed  or  protests  held  in 
defiance  to  the  sediment  treatment  technology? 

#  of  on-site  injuries:  How  many  people  are  at  risk  of  being  physically  injured  during  the 
sediment  treatment  project? 

Rate  of  medical  treatments:  How  many  people  will  seek  medical  treatment  for  ailments 
such  as  cancer,  respiratory  problems  or  digestive  problems  as  a  result  of  the  sediment 
treatment  technology? 

#  of  birth  defects:  How  many  newborn  babies  will  be  infected  with  a  disease  as  a  result 
of  the  sediment  treatment  technology? 

Amount  of  contaminant  disposal:  What  is  an  acceptable  amount  of  contaminants  that 
can  exist  without  causing  any  ill  effects  on  the  San  Francisco  Bay? 

Distance  of  disposal  site  from  residential  area:  How  far  does  the  disposal  site  have  to 
be  to  not  affect  the  San  Francisco  Bay  area? 

Contaminants  in  air:  What  is  EPA’s  acceptable  level  of  contaminants  in  the  air? 

Contaminants  in  soil:  What  is  EPA’s  acceptable  level  of  contaminants  in  the  soil? 

Quality  of  drinking  water:  After  treatment,  will  the  water  be  potable  or  non-potable? 

Migration  rate  of  drinking  water:  Since  there  is  a  possibility  of  contaminant  runoff  in 
the  water,  how  much  of  the  natural  drinking  water  will  be  affected  by  the  sediment 
treatment  technology? 

Availability  for  recreational  boating/fishing:  Will  the  waters  of  the  San  Francisco  Bay 
be  safe  enough  to  permit  recreational  boating  and  fishing? 

Total  cost  of  site  cleanup:  What  is  the  total  cost  for  the  sediment  treatment  technology? 
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Amount  of  total  depreciation:  After  the  sediment  treatment  technology  has  been 
applied,  how  much  will  the  land  depreciate  in  its  market  value? 

#  of  fish/wildlife  deaths:  How  many  fish/wildlife  have  died  as  a  result  of  exposure  to  the 
sediment  treatment  technology? 

! 

#  of  fish/wildlife  poisonings:  How  many  local  fish/wildlife  can  be  infected  with  a 
disease  if  exposed  to  the  sediment  treatment  technology? 

Is  seafood  consumable?  Will  the  seafood  in  the  San  Francisco  Bay  be  consumable? 

Project  completion  time:  How  long  will  it  take  for  complete  restoration  of  the 
sediment? 

Stability  of  foundation  after  treatment:  Has  the  contamination  and  proposed  treatment 
affected  the  stability  of  the  soil  to  support  the  foundation  of  future  building  in  the  area? 

Time  for  site  to  naturally  restore  itself:  How  much  time  will  it  take  for  the  sediment  to 
restore  itself  after  the  treatment  technology  has  been  applied? 

Future  reuse  for  residential  area:  After  completion  of  the  restoration,  will  the  treated 
areas  be  suitable  for  building  a  residential  community? 

Future  reuse  for  businesses:  After  completion  of  the  restoration,  will  the  treated  areas 
be  suitable  for  building  businesses? _ 


Figure  3.4.  Attributes  articulated  by  proxy-RAB  members  for  the  Restoration  Project. 

3.  Delphi  -  Round  1 

At  the  conclusion  of  the  discussion  phase,  where  proxy-RAB  panelists  articulate 
their  relevant  values  (i.e.  attributes),  proxy-RAB  panelists  are  asked  to  participate  in  the 
Survey-Phase  I.  The  panelists  are  instructed  to  rank  each  attribute  in  an  ordinal  manner 
(1  =  most  important,  24  =  least  important),  with  no  ties  allowed.  The  panelists  are  given 
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three  days  to  rank  the  attributes.  Panelists  are  encouraged  to  refer  back  to  the  message 
board  to  understand  each  other’s  opinion  better  before  ranking.^ 

Of  the  seven  proxy-RAB  panelists,  only  four  participated  in  Round  1  of  the 
Delphi.  The  results  are  summarized  on  a  spreadsheet,  where  each  attribute’s  mean  and 
standard  deviation  are  calculated.  The  original  plan  to  use  the  Median  Ranking  and 
Hungarian  Methods  to  aggregate  the  ordinal  rankings  proved  to  be  unsuccessful.  A 
requirement  for  using  the  Hungarian  Method  is  to  have  a  square  matrix,  consisting  of  m 
rows  and  m  columns.  To  achieve  a  square  matrix,  the  original  distance  matrix  (given  by 
the  Median  Ranking  Method)  should  be  supplemented  with  dummy  zeros.  The  small 
number  of  responses  (four)  versus  the  large  number  of  attributes  (24)  calls  for  20 
columns  of  zero  values.  After  appl3ang  the  steps  of  the  Hungarian  Method  to  the  Median 
Ranking  Method,  only  four  attributes  are  distinguished  as  being  more  important  than  the 
remaining  20  attributes;  and  of  those  four  attributes,  no  specific  ranking  could  be 
determined.  The  remaining  20  attributes  could  be  ranked  in  an  arbitrary  manner,  which 
does  not  provide  a  desired  overall  ranking  for  the  attributes.  Therefore,  after  collecting 
the  input  of  each  proxy-RAB  member,  the  mean  value  is  calculated  for  the  rank  given  to 
each  attribute,  and  a  standard  deviation  is  calculated  for  each  attribute  to  determine  the 
distance  Jfrom  the  respective  mean.  The  results  of  the  first  roimd  of  Delphi  are  listed  in 
Table  3.2. 


^  On  the  third  day  of  Round  1,  the  host  received  a  message  that  the  survey  server  would  not  accept  the 
inputs  of  the  proxy-RAB  panelists.  The  survey  was  reformatted  for  the  web,  and  re-published  within  hotirs. 
Due  to  this  problem,  an  extra  day  is  given  for  proxy-RAB  members  to  rank  the  attributes. 
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4.  Delphi  -  Round  II 

In  this  round,  the  final  ordinal  ranking  and  a  summary  of  Delphi  Round  1  is 
published.  One  proxy-RAB  panelist  did  not  provide  justification  for  his  or  her  ranking  in 
Round  1,  and  was  asked  to  provide  comments  on  the  electronic  message  board.  Panelists 
are  encouraged  to  exchange  information  on  the  message  board  to  educate  each  other,  and 
at  times  defend  their  rationale  for  ranking  the  attributes. 

Next,  proxy-RAB  panelists  are  invited  to  participate  in  Delphi  Round  H,  and  are 
given  two  days  to  re-evaluate  their  personal  ranking  with  that  of  the  overall  group 
ranking,  and  provide  a  second  ranking  with  justification.  Of  the  seven  proxy-RAB 
panelists,  five  responded.'*  The  results  of  Delphi  Round  II  are  aggregated  by  determining 
the  mean  and  standard  deviation  of  each  attribute.  The  final  ordinal  ranking  of  attributes 
are  provided  in  Table  3.3. 

F.  RESULTS  SUMMARY 

The  objective  of  this  section  is  to  determine  the  degree  of  association  between 
Delphi  Rounds  I  and  II.  Determining  the  degree  of  association  between  the  rounds  is 
important  because  it  confirms  whether  a  group  is  converging  towards  an  overall  ranking 
of  the  set  of  attributes. 

In  this  thesis  project,  the  mean  value  was  calculated  for  each  round  because  of  the 
difference  in  the  number  of  participants  in  each  round  of  Delphi  (four  panelists  in  the  first 
round,  and  five  in  the  second  roimd).  This  survey  could  not  utilize  the  Nominal  Group 
Technique  (NGT)  because  the  total  for  each  attribute  would  increase  in  the  second  round 


The  Perseus  server  was  down  for  two  hours  on  the  second  day,  and  results  could  not  be  collected  as 
scheduled;  therefore,  extra  time  was  given  for  input. 
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by  a  value  (given  by  the  fifth  panelist),  thereby  providing  no  level  of  convergence  among 
the  attributes. 

To  determine  the  level  of  convergence  of  the  rankings  in  Delphi  Rounds  I  and  II, 
the  correlation  of  the  mean  values  and  standard  deviations  are  taken.  Since  the 
calculation  of  the  correlation  provides  a  number  close  to  one,  it  can  be  summarized  that 
the  mean  ranking  for  each  attribute  in  Delphi  Round  n  are  closely  associated  to  the 
results  obtained  in  Delphi  Round  I.  In  addition  to  the  overall  correlation  obtained  in  the 
average  rankings  for  each  attribute  in  Delphi  Rounds  I  and  II,  the  average  standard 
deviation  associated  with  the  rankings  of  the  attributes  declined  in  Delphi  Round  E, 
indicating  more  agreement  among  the  ranks  given  by  the  proxy-RAB  panelists.  As  a 
crosscheck  to  measure  the  degree  of  convergence  between  Delphi  Roimds  I  and  E,  a  rank 
correlation  test  is  applied  to  the  rank  data,  to  find  Spearman’s  rank  correlation  coefficient 
(R).  The  result  of  0.94  indicates  correlation  in  Delphi  Rovmds  I  and  E,  where  Delphi 
Roimd  I  has  a  positive  influence  on  the  ranking  obtained  in  Delphi  Round  E.  Because  of 
the  influence,  this  thesis  can  conclude  that  after  two  rounds  of  Delphi,  the  proxy-RAB 
panelists  grew  closer  to  consensus,  and  that  further  rounds  would  continue  to  produce 
high  correlation  coefficients  close  to  one.  Table  3.4  provides  the  overall  correlation  of 
the  two  Delphi  rounds,  and  lists  the  convergence  (if  any)  for  the  ranking  of  each  attribute. 
G.  ASSIGNING  WEIGHTS  TO  THE  ATTRIBUTES 

In  Appendix  A,  different  methods  are  explained  to  assign  weights  to  the 
attributes.  As  a  result  of  the  implementation  of  the  Delphi  and  aggregation  methods,  the 
final  ordinal  ranking  of  attributes  are  obtained.  The  rank  sirai  weighting  or  rank 
reciprocal  weighting  methods  are  used  to  assign  weights  to  the  attributes.  Both 
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techniques  are  approximations  where  the  decision-maker  may  choose  his  or  her  method 
of  preference.  Furthermore,  the  researchers  made  the  assumption  that  the  degree  of 
importance  between  each  attribute  is  equal,  as  they  are  ranked  ordinally.  The 
assumption,  although  not  one  hundred  percent  theoretically  defendable,  was  made  to 
simplify  the  process.  The  results  are  thought  not  to  be  affected,  while  the  assumption 
makes  the  process  easier  to  legitimate  to  the  stakeholders.  The  weights  of  the  attributes 
are  calculated  and  listed  in  Table  3.5. 
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Phase  1  -  Survey  Results 


^  ^  ^ 

c 

CO 

(T 


o 

H 

0) 


0>  CM 
CM  CM 
CO 

a>  h. 
^  r>- 

CO  CO 

ir> 


O>C000CMC0h^  CDCMh- 
r^t-ooincM'i-  LoiO’M* 
^COO>t-CMO>  0>t-CM 
'^COCMCDCD^  0)COO 
T-OOi-OCOh-  CMC»CD 
cpT^T-^ 

T-cd^iriioco  cviiocM 


I^C3>CDO)CMCOTroOCDCON.gSI^ 
oxtr^cMincoco-i-inTj-r^Qw 
C^CMO-M-t-ON*  ^  ^  * 

N.COCO^CO'^h- 
m  o>  O)  CM  cj>  o  CO 
o  m  cj>  1-  CO  lo 


in  o 
CM  Oi 


in  Tf  CO 


in  ^  Jg  o 

^  m  in 

CO  m  CD  T-*  SS 

N  ^  0>  CM  ^ 

'M^iricoiriaiincDiri'*”^^ 


0) 

2  ^ 

0) 


in  in  in 
r-  CM 
_  -r^  CO’  CD 

>  T-  1-  CM  1- 

< 


in 

in 


=  N  N 

m'”' 


m 

CM 

in 

in 

o 

in 

h- 

in 

CM 

in 

in 

in 

in 

h- 

in 

in 

CM 

in 

CM 

in 

in 

in 

CT> 

co' 

0) 

00 

00 

W 

CJ) 

in 

CD 

CM* 

h-' 

T“ 

T- 

T“ 

CO 

CO 

CD 

o 

in 

N 

m 

in 

CM 

0) 

o 

T- 

in 

1— 

T- 

CD 

CM 

CO 

T- 

in 

h- 

CO 

CO 

h* 

CD 

CD 

in 

h- 

o 

2 

I 


a 

QQ  ^ 
CC 


o 


in  ^  CO 

1-  CM  1- 


co  'M-  2  00  j::  CM 


O)  O  CO 


CM 


52  f3  in  $i!  CD 


0^  Oi 

CM  ^ 


CM 

CM 


55EC:S;cM2^=Ji!S28  cocDTMncoj;]$25252^ST-cMr^o> 
oc 


m 

CO  T-  o 

< 
cr 


CO  ’M’ 
CM  CM 


cjt^T-oo*® 


< 


CO 

D) 

C 


c  W  -5^ 


CO  c 

£  o 

CO  .52  -D 
CD  O  (D 
X)  cis= 

©  0)  i2 

Cl  d-^ 

.9  .9  <« 

III 

EE- 

o  o  .> 

■D  T3  O 


CO 

0) 

I 

?-  CO 

O 

k. 

w  3 
C  [c 

co  d) 

E  *1 
o  c 
o  o 


CO 

4-* 

c 

0 

E 

■£ 

£ 

IT  w 

CO  "5 

.9  0) 

0  X3 


.CO 

c 

0 

■g 

0  2 

to  _ 

S  £ 

Cl  o 
CO  t: 


0 

'w 

« 


c» 

E 

Ic: 

CO 

*&) 


c 

CO 

c 

ECO 

o 

f  8- 
8!° 

I  8 


0 


•5  o 
0  0 


®C-  oS 
ooooo*5oE.9 


c  c 
0  0 
E  E 
E  E 
0  0 


2  w 

(u  ^ 

'  -  c  c 
o  o 
O  O 


fc-  IS 

0  55 

•ts  o 
0  ^ 

^  0 
o>  c 

^  .E  o 

*5  £ 

>  C  0 

>  -C  0 

0)"0  o 
.E  'fe  0 

2  O 

•=  1-2 
- 

1.2  1 
=  ^  ;= 
§  O)  § 

o  ii  < 


o 

c  t>  « 
0  £  £ 
£0-0 
O  0  0 
<D 

o  *2  £ 

to'S^ 
o  ^ 

O  C  2 

0  9^ 
S  E  o 
I-  < 


«  c^. 
?£ 
*F  -Q 
§  ^ 
0 

*5 

d 

0 


0 

it: 

0  0 
l§ 
-il 
t- 

E  £ 

o  S 
®  5 
P  0 


0 

« 

CD  0 
C  0 
O  ^ 

CO  J?  0 
0  (0  0 

2  T5  .E 

O  0  « 

*s«  0  9 
0  c:  .o 
c  ^  ^ 

O  £  £ 

0  0  0 
•2s  0  0 

2  i  i 
£  £  £ 
®  3  3 

.i  ^  -3 


a.  CO  I-  u.  u. 


58 


Table  3.2.  Results  of  Delphi  Round  I 
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Table  3.3.  Results  of  Delphi  Round  II 
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Table  3.4.  Overall  Results  of  Delphi 
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Table  3.5.  Assigning  Weights  to  the  Attributes 
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IV.  CONCLUSION  AND  RECOMMENDATIONS 


Based  on  the  results  achieved  in  the  implementation  phase  of  this  thesis,  the 
following  lessons  learned  are  identified.  The  lessons  concern  the  participation  level  of 
proxy-RAB  members,  the  articulated  attributes  collected,  and  the  effectiveness  of  the 
internet. 

First,  any  implementation  of  the  internet-based  Delphi  should  not  expect  100% 
participation.  This  thesis  research  originally  planned  to  use  seven  panelists  to  participate 
in  its  survey;  however  only  four  and  five  panelists  participated  in  the  first  and  second 
rounds  respectively.  The  anonymity  feature  of  the  Delphi  method  may  actually  restrict 
the  participants’  accountability  in  contributing  to  the  discussions  and  ranking  the 
attributes.  In  a  typical  Delphi,  participants  gather  in  a  conference  room  and  submit  their 
votes.  While  the  votes  are  anonymous,  there  is  an  assumption  that  each  member  submits 
a  vote,  since  the  effort  to  convene  at  a  physical  location  makes  the  act  of  voting  a  small 
marginal  effort.  Due  to  the  expected  decrease  in  the  participation  rate  over  the  internet,  a 
recommendation  may  be  to  invite  twice  the  number  of  panelists  to  obtain  the  desired 
input  level. 

Second,  since  proxy-  Restoration  Advisory  Board  (RAB)  members  are  used  in  the 
survey,  the  results  might  not  reflect  an  accurate  selection  of  issues  related  to  the  San 
Francisco  Bay  Sediment  Project.  Remediation  technology  selection  is  a  technical  subject 
and,  with  the  exception  of  one  panelist,  the  proxy-RAB  members  do  not  have  the  desired 
knowledge  to  participate  this  type  of  survey  for  reasons  such  as  geographic  location,  real 
sediment  issues  in  the  San  Francisco  Bay  area,  and  the  actual  objectives  of  existing 
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RABs.  Better  survey  results  can  be  obtained  by  using  real  stakeholders,  and  having 
technical  experts  available  to  answer  any  treatment-specific  questions. 

Third,  the  flexibility  introduced  into  the  process  by  using  the  internet  to  facilitate 
the  Delphi  has  positive  and  negative  implications.  The  use  of  the  internet  allows  for 
broader  participation  since  participants  are  not  constrained  by  geographic  locality  or  time. 
However,  this  thesis  research  has  fotmd  the  internet-based  Delphi  method  to  require  more 
time  than  the  traditional  Delphi  method.  Because  participants  rarely  participate 
concurrently,  panelists  are  required  to  visit  the  fomm  more  than  once  to  read  the 
responses  of  others  and  to  provide  any  additional  comments. 

Fourth,  the  level  of  technicality  may  also  influence  the  time  it  takes  to  complete 
the  Delphi  method.  Highly  complex  issues  may  be  easier  to  explain  with  traditional 
methods  such  as  physical  proximity.  If  the  subject  is  technical,  more  time  may  be  taken 
to  educate  and  elicit  values  concerning  the  discussion  topic.  When  the  panelists  meet 
physically,  the  exchange  of  information  occurs  much  faster  than  by  posting  messages  on 
an  electronic  bulletin  board,  and  the  technical  subjects  can  be  explained  better  by  using 
visual  aids  to  make  a  point.  Also,  there  may  be  some  hesitancy  by  some  to  utilize  the 
internet.  Techno-phobia  may  limit  the  usefulness  of  the  internet. 

Online  chatting  is  a  viable  option  to  use  during  the  discussion  phase  of  the  Delphi 
method.  Although  proxy-RAB  members  have  the  ability  to  use  the  chat  feature,  the 
feature  was  not  utilized  by  the  members  in  this  thesis  research.  This  may  be  for  a  variety 
of  reasons  including  time  constraints  and  niunber  of  RAB  participants.  The  time  of  day 
that  proxy-RAB  members  choose  to  participate  may  not  be  the  optimal  time  for  others. 
This  thesis  research  includes  panelists  from  three  different  time  zones  (one  being 
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overseas);  therefore,  the  likelihood  of  the  panelists  being  online  at  the  same  time  is 
difficult. 

The  Delphi  survey  results  show  that  the  Delphi  established  consensus  through  a 
degree  of  association.  The  ordinal  ranking  of  round  two  is  closely  associated  with  the 
ordinal  ranking  of  round  one  and  the  standard  deviation  of  many  of  the  attributes  at  round 
two  decreased  relative  to  round  one.  If  one  or  more  additional  rounds  of  Delphi  are 
conducted,  more  consensus  might  have  been  obtained. 

During  the  creating  of  the  survey  and  gathering  data,  several  difficulties  were 
experienced.  One  of  the  difficulties  was  formatting  the  questionnaire.  Due  to  the  wrong 
formatting,  panelists  were  imable  to  submit  their  responses  for  two  days.  Only  after 
receiving  a  warning  message  from  Perseus  Corporation  were  the  authors  able  to  identify 
the  problem  and  solve  it.  The  survey  server  unexpectedly  ceased  operations  for  several 
hours,  which  the  researchers  have  learned  is  fairly  common  with  the  use  of  free  internet 
tools.  Given  the  limited  experiences  from  this  research,  the  authors  concluded  that  the 
services  of  Free  Web  Service  providers  might  not  be  reliable  enough  to  meet  the 
requirements  of  organizations.  The  use  of  free  internet  tools  and  servers  should  be 
considered  carefully.  Non-government  and  government  institutions  should  have  control 
of  the  server  and  not  depend  on  others  for  full-time  uninterrupted  service. 

At  the  end  of  Delphi  Round  I,  proxy-RAB  panelists  identified  public  awareness  as 
an  issue  that  should  be  considered  in  the  San  Francisco  Bay  Sediment  project.  Since  this 
does  not  relate  directly  to  the  choosing  the  optimal  treatment  technology,  it  is  not 
included  in  the  value  tree.  There  is  concern  by  the  panelists,  however,  as  to  how  the 
public  will  be  informed  about  the  issues,  results,  remediation  technologies,  and  updates 
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on  the  sediment  treatment  projects.  This  point  re-confirms  the  importance  of  this  thesis 
research  and  its  goal  of  including  public  involvement  and  awareness,  and  of  gathering 
public  opinion  into  a  value  tree  that  can  be  used  by  decision-makers  to  make  the  optimal 
decision.  It  also  confirms  that  the  public  does  have  a  concern  with  the  decisions  made, 
and  with  the  gathering  of  public  opinion,  decision-makers  are  better  prepared  to 
legitimate  their  choice  to  the  public. 

The  original  plan  of  using  the  Median  Ranking  and  the  Hungarian  method 
changed  during  the  implementation  phase.  After  numerous  attempts  of  applying  the 
Hungarian  method  to  the  data,  the  researchers  found  that  it  is  impossible  to  use  a  matrix 
that  is  made  up  primarily  of  zeros.  The  Hungarian  method  uses  the  distance  matrix 
derived  in  the  Median  Ranking  method,  and  because  the  Hungarian  method  requires  a 
square  matrix  or  a  matrix  close  to  a  square  matrix,  an  assignment  could  not  be  derived 
from  the  distance  matrix  calculated  in  this  research.^  Therefore,  the  basic  methods  of 
aggregation  and  statistical  methods  including  standard  deviation  and  correlation 
coefficients  are  used. 

The  use  of  the  internet  is  a  viable  method  in  eliciting  public  values;  however, 
there  are  issues  that  should  be  considered.  If  an  organization  wants  to  gather  public 
opinion  from  people  living  in  a  certain  area,  the  organization  may  have  problems  in 
access  control  to  only  allow  local  residents  to  participate.  Another  issue  is  if  the  number 
of  people  involved  is  relatively  high,  then  the  results  should  be  stored  in  the  database. 
This  requires  web  database  interaction,  and  presents  a  new  opportunity  for  thesis  work. 
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See  section  E.3  of  the  Implementation  and  Results  Chapter. 
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APPENDIX  A.  ADDITIVE  UTILITY  FUNCTION  AM) 
MULTIATTRIBUTE  UTILITY  THEORY  (MAUT) 


A.  VALUE  AND  CHOICE 

In  his  book,  “Making  Hard  Decisions,”  (1996)  Clemen  argues,  “the  basic  reason 
for  using  a  utility  function  as  a  preference  model  in  decision-making  is  to  capture  our 
attitudes  about  risk  and  return.”  (p.  530)  Risk  and  return  are  considered  elementary 
objectives  in  decision-making,  and  usually  conflict  with  one  another  when  pursuing  high 
returns  while  minimizing  risk.  Clemen  (1996)  provides  an  example  of  a  mayor  deciding 
whether  to  approve  or  disapprove  a  new  electric  power  generating  station.  The  known 
factors  in  this  situation  include  the  fact  that  the  city  needs  more  power  capacity;  but, 
building  the  new  plant  would  worsen  the  city’s  air  quality.  The  mayor  might  consider: 
health  of  residents,  economic  conditions  of  the  residents,  psychological  state  of  the 
residents,  economy  of  the  city  and  the  state,  businesses,  and  local  politics. 

A  preference  model  can  assist  the  decision-maker  (in  this  case,  the  mayor)  to 
select  a  single  alternative  from  a  set  of  feasible  alternatives.  Consensus  building  could  be 
used,  which  provides  a  simple  way  of  prioritizing  the  issues  in  a  manner,  which  the 
decision-makers  can  support;  however,  more  mathematical  methods  of  aggregation  may 
be  preferred. 

1.  Objectives 

Decision  analysis  has  been  used  successfully  to  construct  and  solve  problems  for 
numerous  situations.  The  analysis  is  based  upon  choosing  a  solution  approach  that  lists 
the  decision-maker’s  objectives,  and  assigns  a  weight  according  to  its  importance.  When 


67 


there  are  two  or  more  conflicting  objectives,  an  additive  preference  model  may  be  used  to 
calculate  a  utility  score  associated  with  each  objective.  These  objectives  are  weighted 
appropriately,  according  to  the  relative  importance  to  each  other,  and  then  added  together 
to  give  an  overall  solution  to  the  problem. 

a.  Fundamental  versus  Means  Objectives 

To  develop  a  structured  decision-making  process,  objectives  for  solving 
the  problem  must  be  defined.  Objectives  can  be  thought  of  as  a  “desired  end-state 
resulting  from  plarming  activity,”  (Kleindorfer,  Kunreuther  and  Schoemaker,  1993,  p.  36) 
and  can  be  labeled  as  either  fundamental  objectives  or  means  objectives. 

Fundamental  Objectives  are  objectives  that  the  decision-maker  wants  to 
accomplish.  Means  Objectives  are  objectives  that  help  accomplish  the  fundamental 
objectives  by  defining  measured  attributes.  For  example,  “working  fewer  hours  may 
appear  to  be  an  important  objective,  but  it  may  be  important  only  because  it  would  allow 
an  individual  to  spend  more  time  with  his  or  her  family  or  to  pursue  other  activities  that 
represent  fundamental  interests,  things  that  are  important  simply  because  they  are 
important.  Thus,  minimize  hours  worked  is  a  means  objective,  whereas  maximize  time 
with  family  is  a  fundamental  objective.”  (Clemen,  1996,  p.  44) 

Attributes  are  important  in  explaining  and  quantifying  the  higher  level 
values  and  concerns,  and  are  defined  as  the  criteria  for  measuring  objectives.  According 
to  Rowe  (1992),  an  attribute  is  the  quantity  measured  on  an  attribute  scale.  Attribute 
scales  are  used  as  a  tool  to  measure  the  accomplishment  level  of  the  fundamental 
objectives  (Clemen,  1996).  For  example,  in  buying  a  car,  minimizing  cost  may  be  an 
objective,  and  measuring  cost  in  dollars  is  an  appropriate  attribute  scale. 
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b.  Objective  Requirements 

Clemen  (1996)  states  that  several  conditions  must  exist  for  objectives  and 
attributes  to  be  effective  tools  in  decision-making: 

•  The  set  of  objectives  (as  represented  by  fundamental  objective  hierarchy) 
should  be  complete.  All  relevant  aspects  of  a  decision  should  be  included. 
If  the  decision-maker  is  not  satisfied  >vith  the  result  or  is  imwilling  to 
accept  the  results,  then  the  set  of  objectives  would  have  to  be  considered 
incomplete. 

•  The  number  (or  set)  of  objectives  should  be  as  small  as  possible.  Too 
many  objectives  can  make,  the  decision  much  harder  to  address.  In  this 
case,  hierarchy  is  significant  because  each  objective  provides  a  useful 
representation  of  objectives  that  are  important  to  the  decision-maker,  and 
each  objective  is  different  from  the  available  alternatives. 

•  The  set  of  objectives  should  not  be  redimdant  meaning  objectives  should 
not  be  repeated  in  a  hierarchy.  If  there  are  redundant  objectives,  the 
optimal  alternative  may  not  be  chosen. 

•  The  set  of  objectives  should  be  decomposable.  The  decision-maker 
should  be  able  to  think  about  each  objective  separately  without  having  to 
consider  others.  For  example,  a  student  finding  an  apartment  to  rent  may 
look  at  attributes  such  as  monthly  rent  and  distance  from  school.  The 
student  can  think  about  these  attributes  separately:  regardless  of  rent,  it 
would  be  preferable  to  live  closer  to  school;  or,  regardless  of  distance,  it 
would  be  preferable  to  have  low  rent.  Therefore,  the  objectives  would  be 
decomposable  into  the  two  attributes  of  cost  and  distance,  which  can  be 
considered  independently.  If  the  attributes  are  related  in  a  way  that  does 
not  permit  decomposition,  then  the  set  of  objectives  has  to  be  changed. 

•  Means  and  fundamental  objectives  should  be  distinguishable.  As  defined 
above,  means  objectives  accomplish  fundamental  objectives  through 
attributes.  If  fundamental  objectives  are  not  well  defined,  there  is  no  clear 
reason  why  a  decision  must  take  place. 

•  Attribute  scales  must  be  operational.  These  scales  must  provide  an  easy 
way  to  measure  performance  of  each  alternative  or  the  outcomes  of  the 
fundamental  objectives.  The  attribute  scale  of  cost  in  dollars  for  buying 
an  economical  vehicle  is  a  good  example  of  this  rule. 
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Conflicting  Objectives 


c. 

When  two  or  more  objectives  conflict,  the  challenge  of  recognizing  the 
differences  between  each  objective,  and  choosing  the  method  that  comes  closest  to 
solving  the  problem  is  difficult.  Through  implementing  trade-off  analysis,  a  decision- 
maker  can  add  subjectivity  by  deciding  how  best  to  trade-off  a  more  important  (or 
dominant)  objective  for  a  lesser  one. 

When  only  two  objectives  conflict,  the  trade-off  analysis  is  not  difficult. 
Each  alternative  could  be  listed  or  ranked  according  to  how  well  it  meets  each  objective. 
By  plotting  on  a  standard  X-Y  graph,  the  decision-maker  can  move  systematically  in  one 
direction,  through  the  alternatives,  switching  one  alternative  with  the  next.  The  benefit  to 
this  method  is  that  once  a  switch  is  made,  the  preferred  alternative  must  be  better,  and 
there  is  no  need  to  go  back  and  compare  to  a  lesser-preferred  alternative. 

2.  ADDITIVE  UTILITY  FUNCTION 

An  additive  utility  model  is  a  way  of  evaluating  conflicting  objectives  by 
calculating  an  overall  utility  score.  The  overall  utility  is  the  "weighted  sum  of  individual 
utility  functions  for  each  fundamental  objective.”  (Clemen,  1996,  p.  561)  For  the 
purposes  of  this  research,  the  additive  utility  model  can  be  used  to  choose  the  best 
remediation  technology.  The  Additive  Utility  Model  produces  an  ordinal  ranking  of  the 

alternatives.  Assume  that  there  are  individual  utility  functions  Ui(xi), . ,Um(Xni)  for  m 

different  attributes  xi  through  Xm,  and  that  each  function  assigns  a  weight  of  0  to  the  least 
important  and  1  to  the  most  important  attribute  for  that  particular  objective.  The  additive 
utility  function  is  defined  as  a  weighted  average  of  these  utility  functions.  (Clemen,  1996) 
Therefore,  for  m  different  attributes  (xi  through  Xm),  the  utility  of  this  outcome  is 
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TJ(X],. .  .,Xfn)  —  kiUi(Xi)  +  ...  +  kmUm(Xm)  ,  OT 
m 

=  IkiUi(xi) 

i=  1 

where  the  weights  (ki , . . .  ,km)  are  positive  and  add  up  to  1 . 
a.  Utility  Curves 

Utility  curves  provide  a  graphical  representation  of  the  utility  score  in 
response  to  a  change  in  attribute  measurement.  Figure  A-1  depicts  three  types  of  utility 
function  curves  that  represent  a  translation  of  an  attribute  into  utility  units.  The  curves 
are  dependent  upon  expert  evaluators,  who  through  subjective  input,  determine  factors 
such  as  the  attributes  to  be  used,  the  range  of  scores  for  each  attribute,  and  the  conversion 
factor  for  a  score  to  change  into  a  utility  score.  The  downward  curve  in  figure  A-l(l) 
represents  a  decreasing  utility  function  for  an  attribute,  meaning,  as  the  attribute  increases 
towards  the  higher  level  of  its  defined  range,  the  utility  score  of  that  attribute  has 
decreased.  For  example,  in  the  sediment  treatment  project,  the  sooner  the  project  is 
completed,  the  better  the  satisfaction  level  of  the  objective.  Therefore,  time  is  the  defined 
attribute,  and  the  experts  of  the  project  have  defined  minimum  and  maximum  total 
project  time  for  each  remediation  technology  to  complete  the  task.  If  the  project  is 
completed  in  minimal  time,  the  utility  assessed  is  high  (closer  to  1);  however,  if  the 
project  takes  too  much  time  and  is  measured  towards  the  upper  range  level,  a  low  utility 
is  assessed,  making  that  particular  sediment  technology  less  desirable  considering  the 
time  attribute  only. 
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(1)  Time 
Attribute 


(2)  Cost 
Attribute 


(3)  Completeness 
Attribute 


Figure  A-1 .  Examples  of  Utility  Function  Curves. 

The  second  curve  (Figure  A-1  (2)),  represents  a  higher-scoring  utility  for  a 
lo\v  measure  of  the  attribute.  However,  as  the  level  of  the  attribute  increases,  the  utility 
score  decreases  sharply  imtil  a  certain  level,  and  after  a  point  the  utility  score  is  constant. 
For  example,  if  the  decision-makers  want  to  complete  the  sediment  project  at  a  low  cost, 
a  higher  utility  will  be  given  to  those  alternatives  whose  cost  falls  within  the  lower  range. 
The  third  curve  (Figure  A- 1(3)),  depicts  a  linear  curve,  displays  a  direct  relationship 
between  an  increasing  attribute  level  and  an  increasing  utility  score.  An  example  of  this 
situation  would  be  the  desired  level  of  completeness  (progression  of  cleanup)  for  the 
sediment  project,  where  the  attribute  moves  towards  a  utility  score  of  one,  as  the  project 
becomes  more  complete. 

b.  Indifference  Curves 

Indifference  curves  graphically  demonstrate  the  set  of  alternatives 
among  which  a  decision-maker  is  indifferent.  In  sketching  indifference  curves,  the 


decision-maker  will  find  that  each  slope  produced  from  the  alternatives  is  related  to  the 
trade-off  rate  that  was  assessed.  For  example,  a  decision-maker  may  be  indifferent 
between  two  sediment  treatment  technologies,  when  considering  the  attributes  of  cost  and 
time.  The  total  utility  is  the  same  for  both  alternatives,  therefore,  the  decision-maker  is 
indifferent  between  choosing  the  two  alternatives  with  respect  to  cost  and  time  only.  The 
slope  could  also  be  defined  as  the  marginal  rate  of  substitution,  or  the  rate  at  which  one 
attribute  can  be  used  to  replace  another  (Clemen,  1996)  (see  Figure  A-2). 

High 

Project 
Time 

Low 

Low  High 

PROJECT  COST 

Figure  A-2.  Indifference  Curve. 

3.  ASSESSING  INDIVIDUAL  UTILITY  FUNCTIONS 

There  are  different  methods  regarding  the  assessment  of  the  individual  utility 
functions  including  the  assessment  of  ratios,  the  calculation  of  proportional  scores,  and 
the  standard  utility  function  assessment.  In  this  appendix,  the  simpler  methods  (ratios 
and  proportional  scoring)  are  explained  to  offer  a  choice  among  determining  utilities  (or 
utility  scores).  For  additional  information,  refer  to  Clemen  (1996). 
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Ratios 


a. 

Ratios  present  a  way  of  determining  utilities  based  on  a  numerical 

comparison  of  attributes,  whether  the  attributes  are  quantitative  or  qualitative.  This 

method  converts  performance  scores  into  utility  scores,  where  the  range  of  the  utility 

scores  are  from  zero  (lowest  level  utility)  to  one  (highest  level  utility)  (see  Figure  A-1). 

Clemen  (1996)  uses  the  following  example:  the  decision-maker  decides  that  color  is  an 

important  attribute  in  purchasing  an  automobile.  Using  the  ratio  approach  for  qualitative 

measures,  the  decision-maker  determines  a  level  of  importance  in  comparing  one 

attribute  to  another.  In  this  example,  the  decision-maker  decides  that  blue  is  twice  as 

good  as  red,  and  that  yellow  is  2  Vi  times  as  good  as  red.  Points  ranging  from  0  to  100 

can  be  given  to  each  alternative  based  on  its  performance  on  the  attribute.  In  the 

example,  if  30  points  were  assigned  to  red,  then  60  and  75  points  would  be  assigned  to 

blue  and  yellow  respectively.  These  results  can  be  calculated  to  reflect  the  conversion  of 

the  highest  and  lowest  performance  scores  to  the  utility  scores  of  1  and  0  respectfully. 

First,  constraints  a  and  b  need  to  be  determined  so  that: 

0  =  a  +  b(30) 

1  =a  +  b(75) 

Solving  these  two  equations  simultaneously  gives 

a  =  -2/3 
b  =  1/45 

Applying  these  scaling  constants,  the  utility  values  for  the  car  color  are: 

Us  (red  car)  =  -2/3  +  (l/45)(30)  =  0 
Us  (blue  car)  =  -2/3  +  (l/45)(60)  =  2/3 
Us  (yellow  car)  =  -2/3  +  (l/45)(75)  =  1 
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The  utility  of  the  car  color  could  be  used  in  conjunction  with  trade-offs  determined  for 
attributes  such  as  price  and  fuel  economy. 

b.  Proportional  Scores 

Proportional  scoring  is  another  method  that  can  be  used  to  assess  utility 
scores.  This  method  assumes  natural  numerical  measures  exist  (i.e.  dollars  and  years). 
The  lowest  and  highest  performance  measures  on  the  attribute  are  assigned  utility  scores 
of  0  and  1  respectively.  The  calculation  in  this  method  determines  the  utility  scores  for 
any  intermediate  performance  measure  that  exists  between  the  lowest  and  highest 
performance  measures. 

The  general  formula  is  used: 

Ui(x)  =  _  X  —  Lowest  Performance  Score _ 

Highest  Performance  Score  -  Lowest  Performance  Score 

X  -  Xj' 

Xi'"-  Xi' 

Both  the  ratio  and  proportional  scoring  methods  provide  a  way  to  calculate 
the  utility  scores  for  performance  measures.  However,  another  need  in  the  additive  utility 
function  is  to  determine  the  weights  of  each  attribute.  Some  of  the  available  methods  for 
assessing  weights  are  explained  in  the  next  section. 

4.  ASSESSING  WEIGHTS 

Weighing  the  attributes  is  necessary  to  execute  the  Additive  Utility  Function. 

Weights  reflect  the  importance  of  each  attribute  relative  to  the  other  attributes.  There  are 

various  methods  to  assess  weights  including  equal  weighting,  weights  from  ranks,  ratio 

weighting,  the  pricing-out  method  and  swing  weighting.  The  simpler  methods  (equal 

weighting  and  weights  from  ranks)  are  explained  in  this  appendix.  In  this  research,  the 
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“Smarter”  method  in  Logical  Decisions  for  Windows™  (1997)  software  is  used  to 
calculate  weights  in  a  simple  manner.  This  computer-based  application  allows  the  user  to 
calculate  weights  more  expediently  than  by  hand,  and  providing  a  user-friendly  platform 
that  automatically  provides  an  overall  assessment  of  evaluation  to  the  decision-maker. 
For  further  information  on  the  other  available  methods,  refer  to  the  writings  of  Clemen 
(1996)  and  Edwards  and  Newman  (1982). 

a.  Equal  Weighting 

Equal  weighting  is  the  easiest  method  to  weight  the  attributes.  The  decision¬ 
maker  simply  assigns  equal  weights  to  each  attribute.  Thus,  every  attribute  has  the  same 
weight.  Equal  weighting  is  sometimes  useful  when  the  decision-maker  does  not  have 
meaningful  weights  obtained  from  the  stakeholders  or  does  not  have  weights  at  all. 
(Edwards  and  Newman,  1982)  In  this  thesis  research,  the  level  of  importance  of  each 
attribute  is  not  the  same,  and  considered  important  in  selecting  the  optimal  sediment 
remediation  technology. 

Weights  From  Ranks 

In  this  method,  the  decision-maker  has  to  have  an  ordinal  ranking  of  attributes 
(listing  the  most  important  attribute  first,  the  second  most  important  attribute  second,  and 
so  on).  A  numerical  weight  is  then  assigned  to  each  attribute  according  to  its  importance. 
These  methods  provide  a  simple  way  of  assessing  weights;  therefore,  the  decision-makers 
can  utilize  these  methods  to  legitimate  their  decision  better  to  the  stakeholders.  There  are 
two  methods  of  doing  assessing  weights  from  ordinal  ranking: 

•  Rank  Sum  Weighting:  Assign  the  largest  number  to  the  most  important 
attribute,  the  second  largest  number  to  the  second  important  attribute,  and 
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so  on,  until  the  list  important  attributes  receives  the  rank  of  one.  Then 
dividing  each  numerical  number  by  the  sum  of  all  number  gives  weights. 

•  Rank  Reciprocal  Weighting:  Assign  the  numerical  value  of  one  to  the  most 
important  attribute,  two  to  the  next  most  important  attribute,  and  so  on. 
The  reciprocal  of  each  numerical  number  is  normalized.  The  normalized 
numbers  are  used  as  the  weights.  (Edwards  and  Newman,  1982) 

C.  CONCLUSION 

The  objective  of  using  multiattribute  decision-analysis  techniques  is  to  construct  a 
model  that  is  reasonable  representation  of  a  decision-maker’s  value  structure.  The 
Additive  Utility  Fimction  provides  a  weighted  sum  of  individual  utility  fimctions  for  each 
fundamental  objective.  There  are  various  methods  available  to  calculate  individual  utility 
scores  and  weights  where  the  decision-maker  may  choose  the  method  of  preference.  As  a 
result,  the  additive  utility  model  enables  the  decision-maker  to  select  an  alternative  from 
a  set  of  feasible  alternatives. 
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APPENDIX  B.  ROLES  FOR  PROXY-RESTORATION  ADVISORY 

BOARD  (RAB)  MEMBERS 


The  following  is  an  example  of  the  roles  assigned  to  each  proxy-Restoration 
Advisory  Board  (RAB)  member.  There  are  a  total  of  seven  roles  assigned,  with  one 
participant  assigned  to  each  role. 

In  your  role,  you  are  a  concerned  public  citizen  (member  of  the  RAB  -  Restoration 
Advisory  Board}.  Think  about  issues  or  concerns  you  may  have  regarding  this  site 
restoration.  Your  input  will  be  used  to  choose  the  proper  technology  to  remediate  the 
polluted  sites  in  the  San  Francisco  Bay  area. 

CONCORD  NAVAL  WEAPONS  STATION  (NWS) 

Background 

1 .  Demographics-  Concord  Naval  Weapons  Station  (NWS)  is  about  35  miles 
northwest  of  San  Francisco.  It  is  surrounded  by  the  city  of  Concord  to  the  west 
and  south  (population  1 16,000);  the  city  of  Bay  Point  to  the  east  (population 
17,000);  and  the  small  town  of  Clyde  (population  600)  to  the  north.  The  size  of 
NWS  is  13,023  acres.  It  is  the  major  Naval  munitions  facility  on  the  west  and,  as 
an  ocean  terminal  facility,  is  used  to  transfer  ordinance  from  trucks  and  rail  cars 
to  ships  and  vice  versa. 

2.  Contaminants  -  Concord  NWS  is  contaminated  with  heavy  metals.  Petroleum, 
Oil,  and  Lubricants  (POLs),  volatile  and  semi-volatile  organic  compounds. 

3 .  Environmental  Risk  - 

a.  Hydrogeology  —  Concord  NWS  is  bound  on  the  north  by  Suisun  Bay  and  on 
the  south  and  west  by  the  city  of  Concord.  Surface  water  and  sediment  is  the 
pathway  of  greatest  concern  due  to  the  interconnection  of  the  Tidal  and 
Litigation  areas.  Groundwater  at  Concord  NWS  is  not  used  for  drinking 
water  due  to  its  high  Total  Dissolved  Solids  content. 

b.  Natural  Resources  -  Suisun  Bay  is  a  transition  zone  between  saltwater  and 
freshwater  ecosystems  and  is  interconnected  to  the  Concord  NWS  wetland 
areas.  This  area  contains  a  diverse  population  of  fish  and  other  aquatic 
wildlife.  The  Bay  is  also  used  for  recreation.  The  upland  and  the  wetland 
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federal  and  state  designated  threatened  and  endangered  specious.  These 
include  the  Salt  Marsh  Harsh  Mouse,  California  Clapper  Rail,  California 
Black  Rail,  Tule  Elk,  and  the  figwort  family  of  plants  including  the  Delta 
Tule  Pea,  and  Soft  Bird’s  Peak. 

c.  Risk-  Four  sites  are  ranked  as  high  relative  risk  in  the  DOD  Relative  Risk 
Site  Evaluation  System  at  Concord  NWS  primarily  because  of  the  threatened 
and  endangered  species  in  the  sensitive  Avetland  areas  and  recreational  users 
in  adjoining  Suisun  Bay.  The  close  proximity  of  NWS  to  the  Contra  Costa 
County  Water  Wells  surrounding  Mallard  reservoir  has  also  contributed  to 
the  high  relative  risk  ranking.  Risk  to  human  health  and  the  environment 
have  been  reduced  due  to  the  remedial  action  for  the  Litigation  Area  Sites. 
Public  Involvement 

1.  Restoration  Advisory  Boards  (RABs)  -  There  are  10  active  RAB  members. 
The  Navy  and  the  regulatory  agencies  have  given  technical  presentations 
during  the  monthly  RAB  meetings. 

2.  Relevant  Issues  -  An  Information  Repository  was  established  at  the  Central  at 
the  Central  Contra  Costa  Public  Library. 

After  reviewiiig  this  page,  please  go  to  the  Delphi  Website  at 

http://www.delphi.com/nps  survey.  There  you  will  read  the  background  information  on  the 

San  Francisco  Bay  Sediment  project,  and  asked  to  go  to  the  Methodology  Information  and 

Instructions  links. 

*  Information  taken  from  DON  Environmental  Restoration  for  FY  1998-2002. 
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APPENDIX  C.  ARTICULATED  RESPONSES  FROM  PROXY- 
RESTORATION  ADVISORY  BOARD  (RAB)  MEMBERS 


The  following  responses  are  the  results  of  the  discussion  phase  of  the  Delphi 
method.  The  seven-day  discussion  phase  resulted  in  five  responses  by  proxy-RAB 
panelists.  The  results  were  then  interpreted  into  measurable  attributes  relating  to  the  San 
Francisco  Bay  Sediment  Project,  and  placed  in  a  value  tree  (see  Figure  3.3  of  the 
Implementation  and  Results  Chapter). 


From:  MERSINl 

To:  ALL 

6.1 


As  a  citizen  of  this  area  I  am  concerned  with  three  major  factors  HUMAN, 
ENVIRONMENTAL  and  COST. 

HUMAN  ISSUES: 

-What  provisions  will  be  taken  to  insure  “Public  Safety”? 

The  Government  will  cover  -What  health  issues?  (Birth  defects.  Cancer, 
Respiratory,  Digestive) 

-Will  public  notices  be  given?  (Newspaper,  Radio,  TV) 

-Will  the  local  community  be  allowed  to  take  part  in  the  planning  of  the 
restoration? 

-What  effects  will  the  area  have  on  our  pets? 

-What  is  the  level  of  contaminants  in  the  water  supply? 
ENVIRONMENTAL: 

-How  long  will  the  clean  up  take? 

-What  is  the  schedule?  (Start,  Completion) 

-How  many  phases  are  planed  for  the  clean  up  and  what  are  they? 

-Will  the  public  be  given  a  map  of  the  contaminant  area? 

-What  environmental  studies  have  been  conducted  and  will  they  be  shared 
with  the  public? 

-What  is  the  acceptable  levels  of  contaminants  in  the  air,  soil  and  water?  Do 
they  meet  EPA  standers? 

-How  long  will  it  take  for  the  earth  too  naturally  restore  the  area? 

-What  harmful  gases  will  be  in  the  air  and  soil? 

-What  is  the  short  and  long-term  uses  for  the  area? 

-What  effects  has  the  contaminants  had  on  the  stability  of  the  soil  to  support 
[the  foundation  of  future  building  in  the  area? 


81 


COST: 

-What  will  it  cost  to  conduct  the  clean  up? 

-Who  will  absorb  the  cost? 

-Will  this  lead  to  a  deprecation  of  the  local  area? 


82 


From:  FETHIYEl 
To:  MEMOCAN  unread 
5.3  in  reply  to  5.1 _ 


What  will  the  facility  be  used  for  in  the  future?  -  This  is  of  primary 
:  importance,  because  it  will  determine  the  amount  of  clean-up  that 
is  required. 

Can  we  transfer  the  land  prior  to  clean-up?  -  This  allows  the  Navy 
to  stop  paying  for  operating  costs  (security,  fire,  etc.)  during  the 
clean-up  effort.  However,  private  landowners  are  very  concerned 
with  their  liability  upon  transfer. 

What  remediation  technology  is  proposed  to  prevent 
contamination  of  the  bay?  -  This  is  the  biggest  environmental  risk, 
yet  because  it  is  a  long  way  off  (slow  flow  of  pollutants  to  the 
bay),  and  it  doesn't  directly  impact  development,  it  may  be  a  low 
priority  to  the  public.  However,  the  Navy  needs  to  plan  for  the 
remediation  effort-whether  it  be  in  situ  or  via  extraction  (e.g., 
bioslmper  technology). _ ■ 
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[ 

From:  PWHITE27 

II 

To:  MEMOCAN 

n 

5.2  in  reply  to  ^ 

Where  will  the  sediment  go  once  it  is  treated? 

Realizing  pollution  of  San  Francisco  Bay  is  possible  what  percentage  chance 
would  you  assign  that  collateral  pollution  may  be  caused  by  the  cleanup 
effort?  Collateral  meaning  airborne  and  waterborne  pollutants  contaminating 
the  bay/air  in  the  surrounding  area  as  the  cleanup  progresses. 

What  measures  will  be  taken  to  ensure  the  safety  and  health  of  the 
surrounding  population? 

What  impact  will  the  cleanup  have  on  the  surrounding  wildlife  and  fisheries? 
How  long  will  the  cleanup  take? 

E 
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From:  VANllO 

3:26  pm 

To:  ALL 

(1  ofl) 

l8J. 

I  am  a  concerned  public  citizen  and  member  of  the  RAB  -  Restoration  Advisory  Board.  I 
have  three  major  concerns: 

1)  How  can  I  be  assured  that  the  Navy  is  accurately  informing  the  public  of  the  hazardous 
materials  stored  (past  and  present)  at  Concord  NWS. 

2)  If  the  groundwater  at  Concord  NWS  is  not  suitable  for  human  consumption,  how  can  I 
be  assured  that  water  fowl  are  not  at  risk  of  chemical  exposure.  What  measures  have  been 
taken  to  ensure  that  water  fowl  safety? 

3)  Likewise,  are  local  residents  that  use  Suisun  Bay  for  recreational  boating  and  fishing 

safe.  Is  it  safe  to  consume  fish  and  wildlife  fi'om  Suisun  Bay  (but  not  safe  to  drink  the 
water)? _ 
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rom:  SERENEDAY 


o:  ALL 


7.1 


1:19  am 


(1  ofl) 


As  a  member  of  RAB,  there  are  3  main  concerns: 

1.  Short/long  term  effects  of  exposure  to  both  known  and  unknown  contaminates  to  people. 

2.  Safety  of  wildlife. 

3 .  Migration  of  the  contaminated  water  to  populated  areas. 

Points: 

Concerns  1  and  3:  This  site  is  a  known  place  for  migratory  fish,  shellfish,  birds,  and 
mammals,  including,  crab  and  shrimp.  Fish  fi'om  this  area  can,  and  probably  will  migrate  to  ^ 
populated  fishing  areas.  There  has  been  no  study  to  show  the  effects  of  human  consumption 
of  such  contaminated  foods.  Additionally,  although  identified  as  a  potential  issue,  there 
have  been  no  studies  to  delineate  exposiue  of  on-site  persoimel  through  direct  contact  with 
both  the  soil  and  the  near  surface  ground  water,  which  is  likely  coimected  to  the  San 
Francisco  Bay.  In  turn,  this  presents  the  further  issues  of  migrating  contaminates.  Lastly, 
the  soil  supporting  the  Island  (TI)  is  man-made  composed  of  dredged  materials  consisting 
of  poorly  graded  "fine"  sand  placed  over  Yerba  Buena  Shoals.  This  will  allow  for  the 
contaminates  to  seap  through  to  the  water.  Additionally,  although  YBI  is  a  natural  rock 
island  with  minimal  soil  cover,  there  is  still  the  issue  of  runoff  contaminates  into  the  water. 

It  is  foreseeable  that  the  contaminates  fi'om  both  of  these  areas  would  transport  potential 
contaminants  to  the  Bay  or  runoff  would  infiltrate  into  the  Franciscan  sandstone  and  shale. 
Concern  2:  It  is  known  that  this  area  hosts  migratory  birds,  mammals,  and  fish.  These 
include  waterfowl  and  passerine  birds,  the  California  sea  lion  and  harbor  seal.  A  survey  of 
both  Federal  and  California  endangered  or  threatened  species  identified  seven  animals  and 
1 7  plant  species  in  the  contaminated  area. 

Overall:  There  have  been  no  studies  showing  what  the  effects  of  how  the  removal  from  one 
site  will  effect  the  water  and  soil.  Where  will  it  be  removed  to?  stored?  Process?  effect  on 
wildlife?  Due  to  the  ongoing  possible  contamination  of  the  water,  it  would  be  more 
amenable  to  the  community  to  have  the  entire  site  cleaned.  This  would  cause  less  disruption 
o  the  wildlife  and  it  would  ensure  as  little  further  contamination  of  the  water  as  possible. 
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